🧪 Experiments

How Recast incorporates experiments

Experiments (or Lift Tests) are a great way to improve your MMM by incorporating extra information. Experiments can come in a variety of forms, anything that produces an estimate of the incremental impact of a spend channel can be included. Some common examples include:

• Geographic Testing (e.g. GeoLift)

• Direct Mail matchback with holdout testing

• In-platform A/B testing

As long as the experiment produces an estimate of the incremental effect of the spend for a specific channel on a specific set of dates (see below for more details on what we need), it can be used to calibrate your MMM. The MMM is calibrated by constraining the prior on the dates of the test. Prior to incorporating the experiment, your channel will have an ROI prior that is constant across time (these values are available on the Configure tab in the Recast app), that looks something like:

Inserting the test will change the bounds around the test dates, so all ROI priors go through the smaller bounds:

This will ensure the final estimate of ROI ends up within the bounds of the test for the test dates. Because ROIs are constrained to change slowly over time, the days immediately before and after will also be close to the ROI we estimated in the experiment. However, the further you get from the test dates, the more the ROI will be allowed to drift from the constrained ROI.

What’s shown on the Experiments page

The Experiments page displays a list of the lift tests Recast has already ingested into your model. Use the Experiments page to see the lift tests you’ve ran and see how the Recast model is incorporating the experiments into its estimates

Each row shows the ROI or CPA* and the standard error for the test. You can see Recast's estimates of channel performance using the dropdown arrow on the right of the lift test bar. Recasts estimates are for the same channel and time period of your lift test.

*Most tests will be recorded in terms of ROI or CPA. However, certain tests will instead record the estimate of the additional revenue (or conversions) caused by the test spend. In those situations, the row will show something like “150 conversions” which are the amount of incremental conversions estimated to be attributable to the increased test spend in the test region.

What Recast needs for an experiment

If you have any experiments you would like to include in your Recast model, please speak with your model building team. They will help ingest the experiment data into your model. The four most important pieces of info to provide are:

• The channel the test applied to

• The dates the test ran

• The point estimate of the ROI or CPA

• The standard error (or confidence interval) for the ROI/CPA

For more information about how to find these, please see the articles below:

• Advice for analyzing a generic experiment

• Advice for analyzing a Recast GeoLift Experiment

Additional Details

Your data science support can typically configure your test from the four pieces of info in the last section. However, the following are additional configuration options available for incorporating tests.

The configuration metadata for each test has the following pieces of information:

• Variable — Which marketing channel does the test apply to?

• Start date — When did the test start?

• End date — When did the test end?

• Point estimate — What was the mean estimate (possibly in terms of ROI, CPA, or Impact)?

• Uncertainty — What standard error should we use?

• Distribution — What statistical distribution should we use for the prior constraint (see more details below)?

• Type — Should the prior apply to the ROI or the MROI?

  • For typical lift tests, the prior should apply to the channel’s ROI. Set the Type to “Average effect”.

  • If the lift test measured the effect of the just last dollar spent, rather than the total effect of the ad spend, then the prior should apply to the MROI. In this case (which is highly unusual) set the Type to “Incremental effect”.

• Time — How should the model take the test’s dates into account?

  • Setting Time to “Cumulative” applies the prior to the sum of the channel’s impact from the start date to the end date divided by the sum of the spend in the channel. This is the preferred option.

  • Setting Time to “Daily” applies the prior separately to every day’s ROI from the start date to the end date. Due to variations in spend from day to day, it is likely that each day has a slightly different ROI, so this may not be a realistic assumption.

  • Setting Time to “Bookend” applies the prior separately just to the ROIs on the Start date and End date.

Available Distributions

There are a few different settings you can use to control how we change the priors on the ROI in the presence of a lift test. There are three options:

1. Gaussian - Changes the ROI prior to be N(point_est, std_error)

2. "Strict gaussian" (default) - Uses a mixture of normal distribution to ensure most of the probability lies within two standard errors of the point estimate.

3. Uniform (this is really uniform-ish)

   1. This is a uniform-like distribution with soft edges. See image illustrating it below.

   2. You still give it a “point estimate” and an “uncertainty”, and the prior is roughly flat between point_estimate +/- 1.96*uncertainty, with 95% of the prior weight in that range and 5% outside.

An illustration of the uniform distribution:

Impact Lift Tests

Certain types of lift test may be better configured not by constraining the ROI, but instead directly providing the estimated incremental impact on the test group (see here for some GeoLift test setups that use this format). However, if instead of providing ROI numbers you want to provide details about the impact on the test group, we also need information about the test group size in order to configure your test.

Motivation

A typical two-cell lift test has a “control” cell in which the channel being tested is turned off completely and a “treatment” cell in which the channel being tested is run ‘as normal’. From this type of test we can get an estimate of the channel’s ROI:

1. Take the cumulative sales from the control cell and scale it up by the ratio of the two cells’ sizes.

2. Subtract that from the sales in the treatment cell to get an estimate of the incremental sales attributable to the marketing channel being tested.

3. Divide that by the spend in the treatment cell to get an estimate of the channel’s incremental ROI.

Here, ‘as normal’ means that the channel is roughly as saturated in the treatment cell as it is in the wider market of interest in the same time frame.

Some lift tests do not always follow this structure, and it can be difficult to extract an ROI number from them that aligns with what an MMM expects. For example:

1. The channel might be on in both the treatment and control cells.

   1. One common test has the channel running at typical spend levels in one cell and increased spend levels in the second cell.

2. In the treatment cell, channel spend may be so high or low that the channel’s saturation in the cell is much different than its saturation in the wider market.

Impact tests give us tools to incorporate these kinds of atypical lift tests into the MMM by taking into account spend levels and saturation.

What do we know about the test?

First, decide which cell will be “cell 1” and which cell will be “cell 2”. The choice doesn’t matter, just make sure to use the same labels from here on.

We need to know:

• Sales in cell 1 minus sales in cell 2

• The standard error for the above difference

• What proportion of spend in this channel typically is spent in cell 1? In cell 2?

• What’s the ratio of spend in cell 1 during this test relative to typical spend in cell 1? In cell 2?

Example A: Doubling Spend Test

Suppose a marketer ran a two-cell test. In cell 1 they doubled spend in the channel relative to normal and in cell 2 they kept spent at typical levels. Cell 1 represents about 25% of the market and cell 2 represents 10% of the market.

These would be the associated values corresponding to the last two bullet points above:

• Cell 1 typical spend proportion: 0.25

• Cell 1 test spend ratio: 2

• Cell 2 typical spend proportion: 0.1

• Cell 2 test spend ratio: 1

Example B: Typical Holdout Test

Suppose a marketer ran a typical holdout test. In cell 1 they kept spend in the channel at typical levels and in cell 2 they cut spend completely. Cell 1 represents 95% of the market and cell 2 represents the other 5%.

These would be the associated values:

• Cell 1 typical spend proportion: 0.95

• Cell 1 test spend ratio: 1

• Cell 2 typical spend proportion: 0.05

• Cell 2 test spend ratio: 0

Note that since this is a typical holdout test we could also calculate an incremental ROI from it. We don’t have to use the impact lift test functionality in this case.

Configuring into Recast MMM

When adding the test into Recast, we will need these values:

1. Type = Impact

2. Point estimate = Sales in cell 1 minus sales in cell 2

3. Uncertainty = Standard error of (sales in cell 1 minus sales in cell 2)

4. Cell1 Typical Spend Prop = What proportion of spend in this channel typically is spent in cell 1?

5. Cell1 Test Spend Ratio = What’s the ratio of spend in cell 1 during this test relative to typical spend in cell 1?

6. Cell2 Typical Spend Prop = What proportion of spend in this channel typically is spent in cell 2?

7. Cell2 Test Spend Ratio = What’s the ratio of spend in cell 2 during this test relative to typical spend in cell 2?

Configure the other fields (dates, distribution, etc.) as usual.