Multi-Stage Modeling

The Multi-Stage Model is a survival model that estimates the conversion rates from the initial KPI to a second stage KPI further down the funnel and allows you to estimate channel performance at driving these these lower funnel KPIs. This is an add on feature to the core Recast MMM. If you’d like to use this for your model, talk to your account rep or data scientist contact at Recast!

1. What Does Multi-Stage Modeling Solve?

Recast MMM excels at modeling top-of-funnel acquisitions, such as “account creation” or “trial signup”, which aren't always revenue-generating. Clients may be interested in how quickly these acquisitions convert to revenue-driving actions, like funding an account or upgrading from a free-trial to purchasing the product. Multi-Stage conversion modeling addresses this by linking the first-stage conversion (e.g., “free trial start”) to a second-stage conversion (e.g., “convert to paid”).

The multi-stage functionality in Recast accounts for not only the average conversion rate between the first-stage and the second-stage, but also accounts for how different marketing channels may drive different conversion rates.

2. What Data Do We Need?

To use multi-stage modeling, Recast needs:

• Historic data as long as the data of marketing activity used for the Recast MMM

• The number of acquisitions made on every day

• The number of acquisitions that convert to the second stage d days after being acquired.

Most clients will provide data in wide format, shown below. We see that on 2022-02-03, 1,303 customers were acquired in the first-stage. On day 0, 264 converted to the second stage. On day 1, 37 converted to the second stage.

Clients can also provide data in long format. Either format is fine, but we will eventually be converting the data to long format when we clean the data.

Important Note on Format:

• Confirm with the client whether the conversion data is cumulative or daily. It's preferable to receive daily conversion counts rather than cumulative totals. If the client can only provide cumulative counts, we need to do some extra data munging to get daily counts.

• Ensure missing values are not replaced with 0s. If conversions for a future date haven't been observed yet, they should be marked as NULL/NA, not 0.

  • For example, if on September 1, 2024, you acquire 1,000 new customers:

    • By September 3, you prepare the data for multistage conversion.

    • On day 0, 100 of the 1,000 users converted to the second stage. On day 1, 50 converted.

    • Since you haven’t observed conversions for days 2, 3, etc., these should be recorded as NA, not 0.

• As a corollary to the above, ensure 0s are genuine counts and not placeholders.

3. The Model

Model Details

The model begins by assuming that customers acquired from each channel, , have their own conversion rate on day , denoted . These rates are “latent”, meaning we can not observe them directly and have to infer them from the data.

Each day, we acquire some “mix” of customers from each channel. We call those users acquired on the same day a “cohort”. The “mix” of customers is provided by the daily shifted impact values from the MMM. For example, we may predict 100 new customers will be acquired today, of which 50 come from direct mail, 30 from facebook, and 20 from youtube. We observe the cohort over time and can determine which proportion of the cohort converts on day after being acquired.

The conversion rates in the cohort are not the same as , since the cohort is a weighed average of acquisitions from each channel. However, since we have an estimate of the weights from the MMM, we can infer the values of using the following model

We model each channel’s conversion rate at time on the logit scale. We estimate a mean conversion rate for organic, , a mean conversion rate for paid channels, , and a channel level additive effect, . The effect of time is modeled as a random effect, , with variance .

We compute the weighted average of the using the weights for cohort and channel . We use as the conversion rate on day since being acquired. The new conversions, , are modeled as binomial random variables with parameter , and number of trials . Here, is considered the “at risk number” — or the number of acquisitions who have not yet converted by time . Once the weighted average is computed, we add a day of week effect.

Cumulative conversion curves, (or survival curves) can be obtained from the . The interpretation of the is similar to the hazard function, so we can compute survival curves as

and cumulative conversion rates as

4. Including Multi-Stage Models in the Dashboard

We can include multi-stage models natively in the Recast dashboard (and planning tools like Goals, the Forecaster, and the Optimizer) by making a few transformations to the data of the primary model.

These transformations are relatively simple; we need to transform the following variables:

1. The spend-response curves for each channel (as well as their resulting CPAs, mCPAs, and Impacts)

2. The baseline

3. The spikes

With the following adjustments:

1. Multiplying each of the above variables by their cumulative conversion rate (or divide, as the case may be). For example, if a channel has a $100 CPA for a “Sign Up” endpoint, and 50% eventually convert to a “Paid Upgrade”, then this channel will have a $200 CPA for the “Paid Upgrade” endpoint.

2. Convolve the shifted-impacts of each variable by the daily conversion rates . A small detail here is that needs to be normalized such that they add up to one, since we have already multiplied by the cumulative adjustment in (1).

With these transformations of the variables in the primary model, the underlying data can be published as if it were a normal Recast model with all of the main features available. Certain sections of the dashboard, such as Backtests, Prior vs. Posterior, and Experiments, will not be available since they will only apply to the primary model.