🌎 Recast GeoLift

Why run a Recast GeoLift test?

Recast GeoLift is an alternative method of estimating the incremental effect of advertising on your business. It can be used to validate an MMM (build confidence that an MMM is providing true estimates) and calibrate an MMM, particularly for channels with large confidence intervals.

What is a Recast GeoLift test?

A Recast GeoLift test is an experiment conducted to find the true incremental value of your ad spend. During a lift test, you change the spend (increase or decrease) in selected geographies and compare the results with the outcome in the geographies where spend was held constant. By comparing the changes in the test and control groups, you can find the ‘lift’ or incremental change in revenue driven exclusively by the ad spend.

When to run a lift test?

Recast GeoLift can be particularly helpful when you are seeing large confidence intervals in your model or when your in-platform estimates differ from your model’s estimates.

By conducting a lift test, you are able to provide your model with real information on the effect of your ad spend. This helps the model anchor its parameters to a ground truth. As a result you should see the confidence interval of your model’s estimates decrease following a GeoLift test.

Moreover, if you are seeing different estimates by your model and in-platform numbers, it can be difficult to take action on the numbers you are seeing. Using Recast GeoLift , you can provide your model with a snapshot of more precise incrementality data to train and improve your estimates. Through the cycle of learning more about the incrementality of each channel and improving your estimates, you can make more confident bets and improve your outcome.

Limitations of Recast GeoLift

Advertising buys must allow geographic targeting – if it’s hard to limit spend inside a geography and there’s spillage to surrounding geographies, the estimates won’t be as accurate.

Recast GeoLift provides point in time estimates – unlike MMM, the performance measured from a Recast GeoLift test applies to a particular point in time. Measuring different points in time would require running a new test. Our Bayesian MMM works well with the Recast GeoLift output as we estimate incrementality for each channel every day. This means we can apply the knowledge gained from the Recast GeoLift model accurately to the specific time when the test was run without making assumptions about marketing performance over time.

Heterogeneous geos- If different states have different demographics who respond differently to ads, the results will be less accurate

Outliers - eg. a major localized event during the test period could skew the results.

Beta Release - Recast GeoLift is currently in Beta. In particular our application may struggle with high levels of traffic. If the app is not responding to input, please try again later.

What can I do with Recast GeoLift ?

Recast GeoLift allows you to both analyze the results of lift tests as well as design optimal lift tests.

How does Recast GeoLift work statistically?

Recast GeoLift uses a statistical method called augmented synthetic controls to help estimate the incremental effect of the experimental change. When running an experiment, you typically want your test and control groups to be as similar as possible (except for the treatment). Synthetic control models help with this by analyzing the pre-test period and assigning each control geography a weight such that when you sum up the conversions in the control group, it is as close to the test group as possible. Then, these weights are used when analyzing the experiment to determine if the change in conversions in the test group was caused by the spend changes or by random fluctuations.

How to design an optimal Recast GeoLift test?

The main goals of the Design tool is to select geographies to include in your test and control groups and get a spend recommendation for the test group.

Step 1: Ingest your data

First we need a dataset of the historical outcome variable for each day for each geography. Geographies can be at any level that you have the ability to target (e.g. states, DMAs, etc). If you use zip codes as the geography, Recast GeoLift can automatically convert these to commuting zones. Your CSV should contain 3 columns:

• Location (geography)

• Date

• KPI

You can use the tool to map the columns in your dataset to the date, outcome variable and location ID columns as well as specify the date format in your dataset.

Here is a sample dataset

Once you have mapped your columns, you can ingest your dataset. You can see your KPI over the time period for your top 8 geographies by volume as well as click through your dataset. Use these visualizations to check for any data problems.

The historical dataset helps the experimentation tool analyze similarities and differences in geographical performance to select the best geographies to include in your test group. Comparison between geographies with similar historical performance results in more powerful statistical analysis as we can more accurately identify the differences as a result of the spend change.

Step 2: Configure your analysis

During the configuration phase, you will provide parameters for Recast GeoLift to work with. Recast GeoLift will use those parameters to run simulations in order to determine which geographies would make up the best test as well as a recommended spend amount to get usable results.

First select your experiment type.

There are two types of experiments:

Spend increase: This type of experiment makes sense when you have extra money and want to increase your KPI, you think you might be underspending in the channel and want to test increased spend, or when you are adding a brand new channel to your mix.

Spend decrease: This type of experiment makes sense when you want to save money (at the cost of some of your KPI), or when you think you might be overspending in a channel.
If you’re not sure which to do you can do the analysis twice and compare the recommended plans for each type.

Next, select the KPI type: revenue or conversions. If your KPI is not either, select conversions if you think in terms of CPA and revenue if you think in terms of ROI.

Then select your experiment parameters.

Approximate Channel ROI/CPA: Your expected ROI/CPA helps Recast calculate the amount of revenue/conversions you can drive in your test group. For example, if you want to spend $10,000 and think your CPAs are around $100, this means we can simulate 100 additional conversions in the test geographies and estimate whether that provides statistically meaningful lift. In general, using a conservative number (high CPA / low ROI) will result in a more conservative test, meaning GeoLift will recommend more dramatic changes and you’ll have more statistical power.

Experiment length: Provide how many days you want to run the experiment. If you’re not getting good results with a smaller number of days, increasing the amount of days may help.

Cooldown period: The cooldown period after the experiment is a time period where you are no longer spending extra in the channel, but you are still observing the test geographies because revenue/conversions are still coming in from the previous spend. Choosing a short cooldown period might cause you to miss conversions/revenue, while choosing a long time period will cause additional noise that makes it harder to estimate precise ROIs.

Approximate $ for test (or to turn off): This is a starting point for how much money you would hope to spend on the test. Recast GeoLift will first take this spend amount and your efficiency expectations and try to find geographies for which this size of experiment produces a meaningful result. If it cannot find any, it will error and you may need to increase the size of this (or up your ROI expectations). Once it has found a set of geographies with good potential, step 3 will do additional simulations to refine the spend recommendation.

Optionally, you can select certain geos to include or exclude in your test geo. If you leave this blank Recast will select the optimal test geo for you. This can be useful if, for example, you cannot increase the spend in certain geos or you want to exclude certain geographies because other changes happening in the geography may confound the experiment. Alternatively, you can use the Exclude Completely to ensure locations are not used in the design of your experiment at all.

Number of test geos to consider: this allows you to place limitations on the number of geographies you test in. By default, Recast GeoLift allows between 2 and half the total geographies, but you can narrow this down. The simulation engine will try different combinations of geos between whatever high and low numbers you put in

Click “Determine test markets” when you are ready. Recast GeoLift will analyze your data and provide options for various experiment configurations which you will be able to select from. Recast GeoLift ranks the experiment options in terms of the difference between the simulated lift and the lift estimated. If Recast GeoLift is unable to find test groups that result in low error and statistical significance, it will fail to generate the candidate test markets and instead recommend tweaking your settings. Increasing the spend amount, lengthening the number of days, and increasing the number of test markets are all ways to increase the statistical power.

Location shows the set of geographies that belong in the test group.

Baseline Revenue/Conversions is the expected amount of revenue/conversions in the test geography in the simulated “business as usual” time period.

Simulated Lift is the additional revenue/conversions the additional investment drove in the simulation.

Estimated Lift is the amount of revenue/conversions that Recast GeoLift attributed to the increase in spend.

Absolute % Error This is a measure of the percent difference between the true simulated lift and Recast GeoLift’s lift estimate. A small absolute % error means that the estimated ROI is close to the true ROI in the simulation.

For each of the experiment configurations, you will be able to see a graph of the expected conversions over time in the test and control geos for the period of the experiment.

You can use this information provided to select a set of test geographies that meets your investment constraints and which minimizes bias.

To get a final spend recommendation for the selected locations and a deep dive into the power at different spend amounts, click “Deep Dive with these locations.”

Step 3: Deep dive power analysis

The results of your power analysis are two testing plans at different effect levels (and different spend levels), as well as an analysis of the likelihood that your experiment results in statistically significant lift.

The two testing plans are a Baseline Confidence Plan that requires less intervention while still meeting the baseline criteria for statistical significance and a High Confidence Plan that will result in smaller confidence intervals. The High Confidence Plan is calculated by multiplying the the baseline confidence numbers by 1.5.

The power analysis graphs below will help you assess the recommended plans. The power analysis runs many simulations for your selected geos in order to help determine how statistically useful the results will be.

How to use the results of the power analysis?

Use the power analysis to:

• Determine whether the experiment configuration is sufficient to detect a statistically significant incremental lift.

• Refine your experiment design to arrive at an experiment that maximizes your chances of detecting statistically significant results.

The power curve shows the probability of detecting a statistically significant effect given the test geos, expected % change in outcome, and duration of the experiment. Statistical significance in this context means our ability to confidently conclude that ROI is not zero. It is not the same as the thing we are primarily interested in, narrowing the size of the confidence interval on the ROI, but the two are related and more power will also mean smaller confidence intervals. 80% power is the minimum we’d recommend for a baseline experiment (what we call the minimal detectable effect).

The next two graphs are helpful in understanding the kind of outcomes to expect in two different scenarios: (1) where the advertising channel has no true incrementality, and (2) where the advertising causes conversions to increase/decrease by the % predicted by following the “High Confidence Plan” (using your ROI/CPA assumptions). In the example pictured below, when there is no true incrementality, GeoLift would estimate a small increase ($561) in the amount of revenue attributable to the advertising channel. The confidence interval would contain positive and negative values for the revenue, and the p-value would be 0.7, meaning we could not conclude that the experiment resulted in any meaningful change. In the example where there is true incrementality, we estimate $25k of additional revenue in the test group ($17.7k - $32.9k confidence interval), and the p-value is 0, meaning we could strongly conclude the experiment resulted in incremental lift. The graphs on the right show how we expect cumulative revenue to progress as the experiment progresses in each of these simulated scenarios.

If the results of the power analysis do not provide you with a feasible testing plan, you can go back to the experiment configuration and tweak the number of geos, extend the duration of the study, or increase your spend amount, or adjust your CPA/ROI assumptions.

How does power analysis work?

Recast GeoLift uses historical data and statistical simulations to conduct power analysis. Here’s how it generally works:
Recast simulates the effects of your experiment. Based on the input parameters you provided while configuring your Recast Geolift experiment, Recast GeoLift runs multiple simulations to estimate the probability of detecting a true causal effect of the spend change as well as the probability of making erroneous conclusions from your results.
Estimate Power: By analyzing the simulation results, Recast estimates the statistical power of the study. It determines the likelihood that the experiment configuration, given your input parameters, will detect an effect in the test geo in comparison to the control geo.

Because the power analysis does a broader range of simulations over more days than the initial test market selection, it may find that the settings you selected are not providing a meaningful test, and will recommend that you increase your spend amount or change other parameters.

How to test an existing design

The Recast GeoLift tool can also be used to test an existing design and simulate the power curve for that design. Testing existing designs might be useful for a variety of reasons:

• You designed the test with another tool and want to check that Recast GeoLift agrees that it’s a good design

• You designed the test awhile ago and want to test the same design with the latest data

• You needed to tweak a design to conform to real world constraints and want to check that it’s still satisfactory

To do this, go through the same steps as designing an experiment, however in the “Approximate $ for test” box put the actual planned spend change.

In the “Markets Required for Test Arm” box, enter all the locations of your previous design:

Then, using the “number of test geos” slider, set both the high end and the low end to the number of markets you input in the previous step.

Click “Generate Test Markets” to run the analysis just for this design. This will produce a spend recommendation that you can compare to the previous design.

Analyze experiment

Once you have conducted a geographic experiment you can upload the results of your experiment and use the Analyze tool to estimate the incrementality of the channel you experimented with.

Step 1: Ingest your data
Simply upload your data in the same way as described in the “Ingest your data” section above. The data should be in the same format of date, location and KPI per date/location. The dates you include should cover the experimental window as well as at least three (and up to twelve) months prior to the experiment.

Step 2: Configure your analysis
To configure your analysis select the following inputs:

• The start and end dates of the test

• The end of the cooldown period

• The locations included in the test geo

• Any controls to exclude (e.g. controls that may have been contaminated with impressions from the test spend)

• The outcome variable type

• The experiment type (spend increase or decrease)

• The spend withheld or added to the test locations

The start and end dates should align with when spend was turned up (or down) and then put back to normal. The end of the cooldown period should be chosen as when you want to stop analyzing the data for additional conversions. The analysis will provide an estimate of the amount of revenue (or number of conversions) attributable to the experiment between the start date and the end of the cooldown period. For example, if you think someone who is influenced by an ad in the channel might not convert for 2-3 weeks, you may try to set the cooldown window 2-3 weeks after the end date. If the analysis window is too short, you may end up undercounting the true incremental effect and making the channel look less effective. If the analysis window is too long, added noise will cause your confidence intervals to grow and reduce the statistical significance of the results. We recommend choosing an analysis window such that the majority (~95%) of conversions you expect to come from the increased spend (or conversely, not happen due to decreased spend) have been observed in the historical data. Your Recast MMM shift curves may be helpful in setting the analysis window.

Step 3: Analyze your results

The analysis looks like the following:

You will be able to see Recast’s estimate of the incremental lift and whether or not your results are statistically significant. These results can be incorporated into your Recast MMM by providing your Recast team the point estimate and confidence interval on the CPA/ROI. Recast does not require “statistical significance” in order to incorporate the experiment results (insignificant results typically just mean larger confidence intervals). For more information about how to configure the experiment into the Recast MMM, see Design, Analysis, and Incorporation Strategies for GeoLift Experiments.

The graphs show the difference in KPI output in the test and control geographies as well as the cumulative output over time in the test and control geographies. Using these graphs you can see the impact of your experimental spend change on the test geo compared to the control geo.

Below, you will see two more pieces of information. One shows the pre-treatment and treatment fit. If the pre-treatment fit is poor, it is more difficult to assign causality to the experimental spend during the treatment period. We want the modeled control and the test groups to be as close as possible in the pre-treatment time period.

Underneath is a table of the control geographies and how large the weight assigned to each. If a heavily weighted control was particularly close to a test region, and you are worried about impression leakage from the test geography to the control geography, you can re-run the analysis after adding the control geography to the “Exclude from control” field.

Glossary

Geo - A set of locations

Test Geo - The set of locations where we will implement a spend change for the duration of the experiment.

Synthetic Control Geo - These are weighted conversions in the control group. We use this to simulate the experiment and determine the probability of significant results.

Lift - The incremental effect of the spend change in the test geo.

Approximate CPA - this is a reasonable guess at the spend required to acquire a customer in the channel of interest. This is used to calculate the total spend required in the experiment to produce the number of conversions required for statistical significance in the test geo. A higher CPA means that you will need to spend more to drive the conversion.

Experiment length - The number of days during which we will implement the spend change in the test geo.

Bias - The simulated difference between the actual incrementality and the effect estimated by the experiment analysis.