2 posts tagged with "SCM"

GeoLift v2.3 Release​

TL;DR​

• The new version of the GeoLift code makes the pre-test analysis (Power Analysis and Market Selection) easier than ever before! The entire process is now handled by a single function: GeoLiftMarketSelection().
• The new package contains dozens of new features, functionalities, and bug fixes.
• A new and improved walkthrough guide is available to help internal and external users run their GeoLift tests from start to finish!

GeoLift v2.3: Power Calculations Made Easier​

Following the feedback we received, we re-built the Power Calculation and Market Selection functionalities from the ground up! The new code makes this process easier, more streamlined, and more powerful than ever before.

What's New?​

• We've streamlined the entire Power Analysis and Market Selection process into a single function: GeoLiftMarketSelection().
• Previously, the process to select the test markets was very convoluted and involved going through several functions: NumberLocations(), GeoLiftPower.search(), GeoLiftPowerFinder(), and GeoLiftPower(). The new function, GeoLiftMarketSelection(), aggregates all of their functionalities and even improves upon them.
• We're soft-deprecating the NumberLocations(), GeoLiftPower.search(), GeoLiftPowerFinder(), and GeoLiftPower(). The development of these functions is now considered complete and they're now superseeded by GeoLiftMarketSelection().
• We've included two new important metrics that can help us make better decisions when comparing between different test market candidates: Average_MDE and abs_lift_in_zero.
• A new and improved ranking system takes into consideration all key model performance metrics to identify the best test markets for a given set-up.
• We've added the option for one tailed tests. These make the GeoLift model much more powerful than it was before, by simply changing the test hypothesis we want to validate. Now you can chose from Positive, Negative and Total tests in the inference section and one sided or two sided tests in the power section.
• The Market Selection process is more flexible and customizable than before! You can now include additional test constraints to focus only on the tests that make sense for our client. These are: the available budget (budget), acceptable holdout ranges (holdout), test markets we want to force into the test regions (include_markets), and markets that shouldn't be considered as eligible test regions (exclude_markets).
• We've added a new plotting method for GeoLiftMarketSelection objects. Through this method you can easily plot different test market selections and compare their model fit, MDE, and power curves! Plus, they lines have the GeoLift colors!
• The new function: GetWeights() makes it easy to save the synthetic control weights into a data frame for further analysis.
• To aid with MMM calibrations, we've included a new parameter in the Market Selection/Power Analysis function: Correlations. Setting it to TRUE allows the user to analyze the similarities between test and the control regions.
• You can also plot historical similarities between test and control regions with plotCorrels().
• A revamped GeoLift walkthrough vignette has been launched to accompany the new version of the package. This new material provides much more detailed explanation on our model, it's parameters, how to run a study, and how to interpret the results.
• We've fixed multiple bug and errors across the package (thanks for the feedback!).
• Thanks to all of these changes, we've significantly reduced the total time needed in pre-test calculations!

Intro to Quasi-Experiments​

While Randomized Control Trials remain as the gold-standard for causal analysis, good RCTs are hard to come by given how complicated and expensive they are to execute. In particular, their reliance on randomization, which is the foundation for their unbiasedness, is often one of the factors that limit their usage. Some of the most common drawbacks of randomly splitting a population into test and control groups are:

• Implementing and maintaining the randomization throughout the experiment requires a robust infrastructure.
• Limiting the treatment to only the test group can be unethical. For example, restraining the control group from receiving life-saving medicine is wrong. This could also be the case for PSAs.
• It is common to have constraints on which units can be part of the test and control groups. These constraints prevent us from having a good randomization. For example, in a geo-experiment there are often a set of locations that need to receive the treatment and some units can’t get the treatment, which severely reduces the possible randomizations and greatly reduces the experiment’s precision.

Quasi-experimental methods offer a great alternative to measure the impact of a treatment (such as an ad campaign) whenever randomization is not logistically feasible or ethical. These methods differ from traditional RCTs in that they don’t use randomization to select the test and control groups. This gives us a lot of additional flexibility in the experimental design at the cost of a typically larger sample sizes and additional modeling assumptions. Nevertheless, under the right circumstances quasi-experiments provide a great alternative to measure a treatment and can empower advertisers that have been historically unable to use incrementality to start taking decisions based on Lift. Moreover, one of the most commonly used type of quasi-experiments are geo tests, in which the units of experimentation are geographical areas such as zip-codes, cities, regions, or states. In this note we will do a historical review of the most commonly used approaches to geographic quasi-experimentation and compare them to GeoLift.