Why did the model results change so much from week to week?

If your model has large revisions in its estimates from one week to another, it may cause difficulty in your ability to build trust and take action on the results. Sometimes revisions are the result of data errors, but sometimes they are simply the result of the model incorporating new information. Below we summarize the most common reasons for large revisions and what to do about them:

Historical changes in the modeled variable or marketing spend numbers

By far the most common cause for large revisions is when the model is told one story on one week and a different story the next week. It’s not uncommon for Recast to receive data like:

These revisions can happen for both the modeled variable (e.g. revenue) or the marketing channels (e.g. Facebook spend). This frequently happens if data is not complete before it is sent (see this related article for why Recast needs complete data).

These changes can have large effects on model estimates of performance because the model is modeling two different versions of the world. One of those worlds was inaccurate, so the estimates from the model were also inaccurate. Since these errors often happen in the recent past, the current estimates of ROI are particularly likely to be effected.

Recast tries to be proactive in alerting clients when we see large discrepancies like this, but if you are worried about large revisions and think this might be the cause, please contact us so we can help assess if that’s the case.

Missing a large promotional event or holiday

We often see large revisions around holidays or promotional events. If we know about upcoming promotions or holidays that materially effect your business, we account for them in the model. However, if we don’t have a forward looking calendar, sometimes these holidays can cause large revisions. To see why, imagine you sell chocolates and Valentine’s Day was last week. You had a huge spike in sales, but because your model wasn’t controlling for Valentine’s Day, it needs to find an alternate explanation for why sales rose so dramatically. It is going to do this by rapidly revising whichever marketing channel seems to provide the best explanation for the rise in sales. This can result in unexpected spikes in ROI.

To alleviate this problem, work with Recast to ensure we have a forward looking calendar of holidays and promotions.

Natural Experiments

The third reason things can revise quickly is not an error, rather it is when the model gets strong evidence that it’s previous estimates were incorrect. This can often happen with natural experiments, like turning off a large channel that has been on for a long time. If you have a channel that has had very consistent spend for the last two years it may be difficult for the model to know if the revenue is coming in is attributable to that channel or is organic. If that spend is suddenly shut off, the model will learn a lot very quickly. If revenue stays consistent, the revenue was likely from organic origins. If revenue drops a lot, the spend was likely causing the revenue. Natural experiments like this can cause the model to reassess past evidence in light of what it learned from the experiment, and often results in strong revisions.