Validating the MMM Forecasts

In order to actually use these forecasts, the whole company needs to be able to trust the forecasts that come out of the model. We’ve found that the best way to build up trust in the forecasts is via two techniques:

1. “Backtesting” the forecasts to show how the model “would have” made predictions historically (this is like how hedge funds test new trading strategies)
2. Demonstrating consistently accurate forecasts over time

Backtests

The backtesting procedure is done at initial model build time by repeatedly re-training the model up to certain points in time and then having the model forecast the data that it didn’t see during the training process.

In order to consistently demonstrate accurate forecasts over time, Recast:

• Has an automated process for running the MMM on a weekly basis
• Saves the model’s parameters each week in a format that can be used for forecasting
• Repeatedly tests that model’s predictions against actuals as you roll-forward in time (i.e. test the 60 day accuracy on the model trained 60 days ago)
• When backtests are accurate, it gives confidence that the model will make accurate forecasts in the future, and that the model parameters (e.g. ROI estimates) are accurate as well.

👍

You can see the results of your backtests under the backtests tab in Insights.

In this screenshot, there are backtests available covering 7, 35, 63 and 91 days.

Different businesses will have different backtest error due to:

• The natural level of “noise” in the business, which depends on industry and KPI selection, among other factors
• The dependence of performance on marketing spend — some businesses are highly driven by external factors such as interest rates or consumer trends that are not predicted by Recast

If you have a very high forecast error or see large changes in your week over week outcome, reach out to the support team at [email protected] and we will help you diagnose and improve your model!