Why are the confidence intervals so wide?

One of the strengths of Bayesian statistical modeling (what we use to power Recast) is the ability to accurately measure uncertainty, which we represent with confidence intervals. Many other techniques use ad-hoc, unreliable methods for calculating confidence intervals that end up understating how much uncertainty there is. In a world as complicated as marketing with quickly evolving ROIs, changing consumer preferences, and uncertain time lags between marketing and purchases, there is a lot of uncertainty.

There are some common issues that will cause wide confidence intervals:

• Raising and lowering spend in multiple channels simultaneously - if you always raise spend on TikTok and Youtube at the same time, the model is going to have a hard time figuring out if Youtube or TikTok is driving more conversions and those channels will have wide confidence intervals.
• Spending only during promotional campaigns - for example, if you only use TV to advertise your big Memorial Day sale, the model will have a hard time distinguishing whether the TV spend or the promotional discount was the reason your sales went up.
• Constant spend levels - if you have a channel where you always spend the same amount the intervals will be wide because the model will have a hard time distinguishing if the conversions are organic or caused by that spend. Rapidly raising and lowering spend is the best way to help the model gain confidence in what’s happening.
• Low spend - the larger the channel the easier it will be for the model to attribute conversions to the marketing spend.

One thing to note is that it’s easy to look at a wide confidence interval and say “oh no, we didn’t learn anything about the channel.” That is not necessarily the case however. For example, suppose you have a linear TV channel and a streaming TV channel, but you always raised and lowered spending in them at the same time. The model will have a hard time knowing if:

(a) Linear is good and streaming is bad

(b) Streaming is good and linear is bad

(c) They’re both about average

This means the confidence interval will be really wide on the channels separately, but combined we can be quite confident that the two channels are just average. Other tools (like the optimizer and forecaster) can help you make decisions about how much to spend in each of these channels.