The Variance Decomposition report

Recast digs into your variance to learn about why there may be a high variance in some of your channel estimates. This is visualized in the variance decomposition report. A high variance may be due to lack of spend, or correlation with another channel, spikes or the intercept.

Technical summary

Setup: for the selected time-period, we sum the daily impact estimates for each channel for each draw to get a 500 x n_channels matrix of (say) total monthly impact.

Regression model: for each channel in turn, we fit a multiple linear regression model (calling lm()) with that channel as the dependent variable and all other channels and the intercept as predictors, with no interaction terms. This models the variation in the dependent variable’s posterior draws for monthly (or whatever) impact as a function of the variation in the posterior draws for the other channels.

Relative importance variance decomposition: for each channel, we obtain the LMG variance decomposition. The LMG method refits the regression model for every possible ordering of predictors and averages R^2 values for each predictor over all the permutations. This gives a decomposition of the variance of the posterior draws of the channel into non-negative contributions from each other channel + leftover “unexplained variation”.

Interpretation: our interpretation of the output should be either narrow or framed as a starting point for further investigation. From this report alone, we don’t even know if we’re particularly uncertain (in absolute terms) about any given channel.

Suppose the contribution of channel A to the uncertainty in channel B’s total impact is large; then we can say something like “if we are uncertain about the impact of channel A in this time period, we are also uncertain about the impact of channel B” or “we can explore reducing uncertainty in channel A as a way to improve certainty in channel B, but we don’t know from this report alone if that’s a promising approach”.