Exogenous Variables

Brands often raise the question “should we control for x?” where x is some exogenous variable that impacts their sales. Things like:

• Macroeconomic factors
• Covid
• Changes to their website
• Competitors pricing
• Etc.

The answer here generally is “maybe.” First things first, we want to think about what questions we want to be able to answer and how we want to be able to interpret the model results. In general, for most of these factors we believe that they will have their impact in two different ways:

1. Impacts on “base sales” (i.e., the intercept)
2. Impacts on marketing effectiveness (i.e., through the effectiveness of marketing)

Because of this, “controlling for” a feature like Covid by treating it like it were any other marketing channel is going to be wrong.

For the most part, for backwards-looking inferences all of these factors will get incorporated automatically into the results via the time-varying nature of both the intercept as well as the channel effectiveness estimates. That is, if Covid had an impact on a business it probably happened by impacting base sales (which will get picked up by the time-varying intercept) and by making marketing effectiveness more or less performant (which will get picked up by the time-varying channel efficiency estimates). This will happen without explicitly controlling for Covid in the model.

Since accurate inferences don’t require explicitly controlling for these variables we generally recommend against controlling for them in the interest of model parsimony.

Contextual Variables

There are some cases where it might make sense to include these exogenous variables in the model. Particularly, we think it makes the most sense to include these variables when we have the ability to predict the value of these variables into the future. So things like “the weather” often don’t make sense to include, but if your business is highly dependent on a variable like “new housing starts” and you feel your organization has the capacity to predict this variable into the future, then it might make sense to include that variable in the model.

The tradeoff of course is that if we include the variable in the model, then we need to be able to predict the variable in order to be able to run forecasts and optimizations into the future which, after all, is one of the most important uses of Recast.

One good use case for contextual variable is for something like a price or offer change. For example, if you increase the price of your product from $5 to $7 then we can include the price change as a contextual variable in order to be able to estimate how that price increase impacted both your base sales (through the intercept) as well as your marketing performance (via the ROIs on your marketing effectiveness). Since this variable is controlled internally it’s very easy to forecast and will make our forward-looking forecasts and optimizations more accurate (since the change in performance due to the price change is important contextual information).

Non-Paid Media Variables

There are other types of marketing activity that don’t exactly look like typical paid marketing activity. These types of activity might include:

• Sales activity like phone calls or texts or site visits from field reps
• Organic content like videos posted to Youtube
• SEO content
• etc.

Whether or not it makes sense to include these variables in the model will depend on a number of factors:

1. Do you have accurate data on this phenomenon?
   1. If you don’t have accurate data that we trust then we shouldn’t include it in the model
2. Can you convert the phenomenon into a currency?
   1. If we can’t convert the phenomenon into a currency then making comparisons between activity-types gets a lot more difficult and it makes “optimizations” impossible.
3. Can you forecast the phenomenon?
   1. If you can’t forecast this phenomenon then you won’t be able to use the forecaster / planning functionality in the Recast platform
4. Are these activities happening at the same “level” of the conversion funnel as marketing activity?
   1. If the activity is downstream of marketings (like cart abandon-emails) then it probably doesn’t make sense to include it in the model since it’s not operating at the same level as marketing spend

First things first: sometimes people want to include every possible business activity in the MMM model. Things like engineering headcount or website checkout changes. In general we don’t think it’s wise to include things like this in the MMM, but rather it’s best to keep the MMM as focused as possible on measuring the performance of marketing activity and not trying to measure the performance of every possible business activity.

Otherwise, if you have variables that are part of your marketing mix and check the boxes above, then it definitely could make sense to include them in the model. Site visits by sales people and handing out samples for pharmaceutical products fall into this bucket and are good candidates for inclusion in the model.

Promotions

UNDER CONSTRUCTION

Modeling Channels without Vary Spend Over Time

Some clients invest in tactics like sponsoring a sports team or having their brand featured on a radio show for a prolonged period of time. In a similar vein, they may also pay a PR firm a fixed fee to generate them organic press. These types of channels represent unique challenges for Recast give we rely on variation in spend over time to understand the relationship between a channel and the client’s dependent variable.

Depending on the data available from clients on for these channels, we have identified 2 options for modeling them:

Option 1: Non-Spend Channel using Impressions

If impressions generated by the fixed spend are available by day or by week historically we can model it as a non-spend channel. The benefits of this are it captures the changing impressions over time, and then also allows the efficacy of the impressions to change over time, saturate, and have a time shift. The cons of this approach are that it can’t be represented in the optimizer, so will have to be factored in outside of this and some of the reports are not available for non-spend channels.

Option 2: Non-Spend Channel Using Binary Indicator Variable

If a variable impressions metric is not available historically, utilize binary indicator variables as Non-Spend channels to indicate when there are events that would drive impressions. This requires that the channel is not always-on, so there are some days we can observe with 0s.

Option 3: 😢

If historical impressions metric is not available, and the channel is truly always on, we won’t be able to estimate the efficacy of the fixed spend channel distinctly from the intercept.

Lift Tests

Revenue Lift Tests

Revenue lift tests measure how much additional revenue was driven by the tested advertising, then use that to estimate the marketing’s ROI.

A control group that wasn’t exposed to the tested advertising can be used to estimate how much baseline revenue would have come from the exposed group had they not been exposed to the tested advertising. We attribute the remaining revenue to the advertising.

Calculating the ROI point estimate

A client gave us this information about a lift test they ran:

Exposed group

Unique Impressions: 9.2 million

Revenue: $97,365

Ad spend: $80,550

Control group

Unique Impressions: 3.7 million

Revenue: $18,735

We estimate the baseline revenue—the revenue that wasn’t driven by the tested advertising—in the exposed group by taking the revenue received from the control group and scaling that based on the relative sizes of each group. For example, if the exposed group was twice as large as the control group, we would expect twice as much baseline revenue in the exposed group compared to the control group.

By this logic, we estimate that there was

$18,735 x (9.2 / 3.7) = $46,584

of baseline revenue in the exposed group, and therefore

$97,365 - $46,584 = $50,781

of incremental revenue in the exposed group—revenue that we attribute to the tested advertising.

Dividing the incremental revenue by the ad spend gives the point estimate for the incremental ROI:

$50,781 / $80,550 = 0.63.

Calculating the ROI standard error

Clients will rarely be able to provide enough information about a revenue lift test to calculate a standard error for the ROI.

When we have had enough information to estimate the standard error, we have often found it to be between 5% and 15% of the ROI point estimate. So without any better information it’s reasonable to use a standard error of 10% or 15% when inserting the lift test into the model.

So when can we estimate the standard error? Ultimately we need some idea of the variance of the revenue.

For one lift test, a client actually gave us the granular per-order data. From this data we could get the population sizes, the numbers of orders, the average order values, and the standard deviations of order values for the control and exposed groups. We made a calculator to put all this info together to get the ROI standard error.

Technical details of the calculator

Here we’ll derive the formulas in the calculator for the standard errors for the revenues.

Our model for purchases is that the revenue for each purchase Xi is drawn from a distribution with mean μ and standard deviation σ, and that that the number of purchases _n follows a Poisson distribution with parameter λ.

If Y is the total revenue, then

It follows from the central limit theorem that approximately

Then, by the law of total variation,

Next we just substitute the estimates for λ, μ, and σ:

• for λ, use the number of orders,
• for μ, use the average order value, and
• for σ, use the standard deviation of the order values.

We calculate the control group and the exposed group standard deviations separately like this, then scale the control group’s standard deviation by the ratio of the two groups’ populations to get the standard deviation of the scaled control revenue.

Finally, the scaled control revenue is subtracted from the exposed revenue to get the point estimate for the incremental revenue, and their variances are added to get the variance of the incremental revenue. Then just divide the point estimate and standard deviation for the incremental revenue by the ad spend to get the point estimate and standard deviation of the incremental ROI.

ROI lift test priors in Recast’s airtable

When we add an ROI lift test in airtable, we include a mean ROI (point_estimate) and a standard error (roi_std_err).

The model then applies a prior to the channel’s ROI of the form

normal(point_estimate, roi_std_err)

Conversion Lift Tests

Conversion lift tests measure how many additional conversions were driven by the tested marketing, then use that to estimate the marketing’s CPA.

Calculating the CPA point estimate

Calculating the CPA point estimate basically follows the same logic as calculating the ROI point estimate.

A client gave us this information about a lift test they ran:

Exposed group

Population: 458,912

Conversions: 4,278

Ad spend: $225,432

Control group

Population: 15,000

Conversions: 72

We estimate the baseline conversions in the exposed group by scaling the conversions from the control group based on the relative population sizes. So we estimate that there were

72 x 458,912 / 15,000 = 2,203

baseline conversions in the exposed group. The remaining

4,278 - 2,203 = 2,075

conversions are incremental and attributed to the advertising.

The incremental CPA point estimate is then

$225,432 / 2,075 = $108.64

Calculating the CPA standard error

Calculating a standard error for the CPA is much simpler than for the ROI, since the only source of variation is the number of conversions. But there is a hack we’ll need to use at the end to get something that makes sense…

Suppose that the numbers of conversions in each group are Poisson distributed. Then the variances of the conversions in the exposed and control groups are 4,278 and 72, respectively.

The variance of the scaled control conversions is therefore

72 x (458,912 / 15,000)^2 = 67,392,

and so the standard deviation of the incremental conversions is

sqrt(67,392 + 4,278) = 267.7.

Now comes the hack.

The 68% interval for the ROI in terms of incremental conversions is

2,075/$225,432 ± 267.7/$225,432.

To get an estimate for the CPA standard error, we take the top end of this ROI interval, invert it to get the bottom end of the CPA interval, then subtract that from the CPA point estimate:

$108.64 - 1/(2,075/$225,432 + 267.7/$225,432) = $12.41

Here’s an R function that will do this calculation for you.

calc_cpa <- function(spend,
                     control_pop,
                     control_conversions,
                     exposed_pop,
                     exposed_conversions) {
  scaled_control_conversions <- control_conversions * exposed_pop / control_pop
  scaled_control_var <- control_conversions * (exposed_pop / control_pop)^2
  
  incremental_mean <- exposed_conversions - scaled_control_conversions
  incremental_se <- sqrt(exposed_conversions + scaled_control_var)
  
  cpa_mean <- spend/incremental_mean
  cpa_se <- cpa_mean - 1/(1/cpa_mean + incremental_se/spend)
  
  list(
    cpa = round(cpa_mean, 2),
    se = round(cpa_se, 2)
  )
}

calc_cpa(
  spend = 225432,
  control_pop = 15000,
  control_conversions = 72,
  exposed_pop = 458912,
  exposed_conversions = 4278
)
# $cpa
# [1] 108.63
# 
# $se
# [1] 12.41

CPA lift test priors in Recast’s airtable

When we add a conversion (CPA) lift test in airtable, we include a mean CPA (point_estimate) and a standard error (cpa_std_err) with units in dollars.

The model then applies a prior to the channel’s ROI of the form

normal(1/point_estimate, roi_std_err)

where roi_std_err = 1/(point_estimate - cpa_std_err) - 1/point_estimate.

Calculating a CPA standard error from an asymmetric confidence interval

Clients will sometimes give us asymmetric confidence intervals for their lift tests. We need to convert those to a standard error for airtable.

If you invert all of the numbers then you may get a symmetric(ish) ROI confidence interval. This is good because the lift test priors we use are symmetric on the ROI scale.

For example, a client sent us test results with a CPA point estimate of $188 and an 80% confidence interval of ($150, $253). You can check that (1/253, 1/150) is roughly symmetric around 1/188.

The R function below converts the original asymmetric confidence interval into a standard error that gives an approximately correct symmetric distribution on the ROI scale after the 1/(mean - se) - 1/mean transformation is applied.

For the example above, we use it like this:

cpa_ci_to_se <- function(point_estimate, ci_width, lb, ub) {
  normal_quantile <- qnorm(0.5 + ci_width/2)
  
  se <- c(
    point_estimate * (point_estimate - lb)/(lb * normal_quantile + point_estimate - lb),
    point_estimate * (ub - point_estimate)/(ub * normal_quantile + ub - point_estimate)
  )
  
  # the SEs resulting from each bound might be different, so pool them
  sqrt(mean(se^2))
}

cpa_ci_to_se(
  point_estimate = 188,
  ci_width = 0.8,
  lb = 150,
  ub = 253
)
# 31.21283

So the standard error we should put in airtable is $31 for this lift test.