The mROI-Similarity Prior | Recast MMM Knowledge Base

We’ve added a structural prior to the model which pools the mROIs of all spend channels. It essentially encodes the prior that, at every timestep, all spend channels’ mROIs should be similar.

This helps prevent the model from giving massively different reads on channel efficiencies and saturations. It kind of acts like an all-channels grouping.

It’s on by default for new models. To enable it for existing models, set the OtherParam use_mroi_similarity_prior = 1.

In addition to pooling the mROIs, for models with use_new_roi_priors = 1, we also automatically calculate spend levels when using this prior, allowing models to adapt to changes in spend and saturation over time.

Theory

The core idea is that we are essentially applying a prior that, to some degree, the business is spending optimally across channels.

But what does “spending optimally” mean? Well, in theory, each channel should be spending roughly at its break even point with profit. If it spent more, that spend would be less efficient, would drive less revenue, and would lead to lower profit. And same for the other direction - if it spent less, you would be leaving efficiency on the table, and you would likewise make less profit.

Concretely this means that effect of the next dollar of spend in the channel (the channel’s marginal ROI) equals the overall profit margin. So, spending optimally across channels means that all channels’ marginal ROIs equal the same value, the profit margin.

This prior encodes a slightly weaker statement, merely that all channels’ marginal ROIs are just somewhat similar. So we don’t need to know the precise profit margin, and we still nudge the model to explore solutions which are more in line with the business’ learned wisdom, reflected in how they have distributed spend across channels.

What gets pooled?

We’ll start by describing an oversimplified version of the prior before getting into the little details.

Suppose we calculate the mROIs for each spend channel and store them in a vector channel_mroi. We want pull these toward each other in the model.

We want to pull larger channels more strongly than smaller channels, to give smaller channels more freedom. So suppose we’ve also chosen a vector of standard deviations that are smaller for larger channels and larger for smaller channels. We’ll use these to control how much the channels are pulled toward the center.

For the prior, we define a free parameter log_mroi_center, and pool the log-mROIs toward that:

for (ch_id in 1:n_channels) {
  log(channel_mroi[ch_id]) ~ normal(log_mroi_center, sigma[ch_id]);
}

That’s basically it. All of the complexity essentially comes from the details:

Q: There’s not just one channel mROI - it changes over time. So what do we actually put the prior on?
- A: The mean squared difference from the per-timestep center.
Q: How do we handle lower funnel channels?
- A: We give them an offset center.
Q: How do we calculate the sigmas?
- A: Based on the channels’ spend levels.
Q: Do we just use the raw spend when calculating the mROI?
- A: No, we smooth the spend first.
Q: How do you calculate the spend levels?
- A: From the smoothed spend.

The penalty is on the mean squared distance from the center

Supposing we’ve calculated the mROI at each timestep for each channel, rather than applying the normal penalty separately at each timestep, we apply it once to the average squared difference between the log-mROIs and the centers.

In Stan pseudocode:

matrix[n_timesteps, n_channels] log_mroi;
vector[n_timesteps] funnel_specific_center;
for (i in 1:n_channels) {
  mean((log_mroi[,ch_id] - funnel_specific_center)^2) ~ normal(0, sigma[ch_id]);
}

This essentially says that the channel should be near the center on average.

A couple notes:

The prior is only applied on timesteps when the channel has nonzero spend, and at least 3 channels together have nonzero spend.
The prior is not applied to non-spend channels.

Note that we have a funnel-specific center in the pseudocode. Upper funnel channels get a different center than lower funnel channels, as described below.

Lower funnel channels get an offset center

Since lower funnel channels will generally have a different mROI from upper funnel channels, we allow the center to be a bit different for the two groups. We define a single parameter, lf_offset, and use that to define the lower funnel center.

In Stan pseudocode:

vector[n_timesteps] uf_center;
real lf_offset;
vector[n_timesteps] lf_center = uf_center + lf_offset;

lf_offset ~ normal(0, mroi_similarity_lf_offset_scale);

This allows the lower funnel channels to have a different mROI center which still shares its shape with the upper funnel center.

Calculating the sigmas using spend levels

The sigmas are calculated according to channel spends. Larger channels get smaller sigmas, more tightly constraining them to the mROI center.

In R pseudocode:

p <- channel_spend_levels / sum(channel_spend_levels)
sigma_min + (sigma_max - sigma_min) * (1 - p)^10

Well, actually, we don’t use the final spend level that goes into the model here, since that’s been adjusted a bit by applying a spend level floor and the spend level multipliers. So technically here we use the pre-adjusted spend levels which better reflect the raw channel sizes.

Smoothing the spend for the mROI

The day-to-day mROI of a channel will change a lot just depending on variation in the spend, so if we want a prior on the mROI that changes smoothly over time, we need to smooth the spend that we use to calculate that mROI.

We smooth the spend by applying a 35-day rollmax (to capture what the spend is when it’s ‘on’, e.g. for direct mail channels) followed by a 231-day rollmean to smooth it out A LOT.

Calculating the spend levels

We calculate the spend levels by taking the mean of the smoothed spend. When a channel has no smoothed spend - because it’s only on at the very start or end of the time period - we default to just taking the median (divided by 2) of the raw spend.

We also apply a spend level floor. When the calculated spend level falls below the specified floor, we replace it with the floor, and scale up the smoothed spend to match.

The spend levels in airtable are completely overridden when use_new_roi_priors = 1. Now, if you want to adjust the calculated spend levels to increase or decrease the saturation induced by this prior, you can use the new mroi_similarity_spend_level_multipliers and mroi_similarity_global_spend_level_multiplier OtherParams. See the ‘control knobs’ section below for details.

Note that, while these spend levels aren’t used when use_new_roi_priors = 0, the smoothed spend is, and so the multipliers still affect the mROI of individual channels.

The calculated world-scale spend levels are stored in stan_data$reference_spend_world and stan_data$lower_funnel_reference_spend_world to help modelers diagnose if something goes wrong.

OtherParameters (control knobs)

These OtherParameters allow you to control various aspects of the prior.

use_mroi_similarity_prior - controls whether the prior machinery is on or off (including whether we automatically calculate the spend levels)

mroi_similarity_spend_level_multipliers - A named list allowing the modeler to scale the smoothed spend used in the prior and the calculated spend levels (& thus change saturation priors) for specific channels. Example:

# only needs to contain channels you want to change
mroi_similarity_spend_level_multipliers = list(
  meta_prospecting = 1,    # no effect, no need to include this
  meta_retargeting = 0.5,  # more saturation
  influencer = 2           # less saturation
)

mroi_similarity_global_spend_level_multiplier - A global spend level multiplier that applies to all channels. Stacks with the channel-specific multipliers above.
mroi_similarity_min_spend_level - The minimum calculated spend level. (Default: 1000)
mroi_similarity_sigma_min and mroi_similarity_sigma_max - Bounds for the sigmas used in the prior. (Defaults: 0.1 and 0.4)
mroi_similarity_lf_offset_scale - The prior scale of the lower funnel center offset. (Default: 0.2)
mroi_similarity_center_level_tau - The prior scale of how much the center can differ from the midpoint of the model’s prior mROIs. (Default: 0.3)
mroi_similarity_center_time_tau - The prior scale of how much the center can vary over time. (Default: 0.5)
mroi_similarity_channel_excludes - A vector of channel names you want to exclude from the prior. For example: mroi_similarity_channel_excludes = c("ott", "podcast")

Notes for modelers

When use_new_roi_priors = 1, this feature sets saturation priors in exactly the same way as before, except that we now calculate the spend levels automatically. So if you want to adjust saturation (either across the board for all channels or for individual channels) you can essentially do it the same way as before, by adjusting the spend levels, only now by supplying multipliers (via mroi_similarity_global_spend_level_multiplier or mroi_similarity_spend_level_multipliers) that then get applied to the automatically-calculated spend levels.

You can inspect the calculated spend levels in stan_data$reference_spend_world and stan_data$lower_funnel_reference_spend_world.

Again, when use_new_roi_priors = 0, the spend levels in airtable are still used to calculate the kappa priors.

Possible gotchas

This feature makes the mROIs of all channels similar, which may produce undesired results in models where some channels are deliberately configured to have different effectiveness. If you have lift tests in the model, check to make sure the model is still roughly consistent with them. Options:
- Weaken the prior by increasing mroi_similarity_sigma_min and mroi_similarity_sigma_max, say to 0.2 and 0.5, respectively, or higher.
- Use the spend level multipliers to change the saturation levels for those different channels.
- Remove the channels from the prior using mroi_similarity_channel_excludes.
- Disable the feature.
You do still need to enter spend levels in airtable (we still use them) in these situations:
- When use_new_roi_priors = 0, regardless of whether or not you’re using this feature.
- When use_new_roi_priors = 1 and you’re not using this feature (use_mroi_similarity_prior = 0).