Mean-Zero Gaussian Processes: An Overview

One available option for the MMM’s time-varying parameters are our so-called mean-zero Gaussian processes.

Straight to the definition

We’ll explain the definition of a mean-zero Gaussian process (GP) by contrasting it with a typical GP.

Typical Gaussian processes

A typical (non-mean-zero) GP can be defined as

where is a parameter vector with one entry for each time step and is a covariance matrix built from the GP’s kernel function.^[1] We’ve used a prior mean of zero here for simplicity.

Instead of sampling from that multivariate normal directly, we can first calculate the Cholesky decomposition , where is a lower triangular matrix with positive diagonal entries and is its transpose. Multiplying that matrix against a vector of parameters sampled from independent standard normal distributions is equivalent to sampling from the above multivariate normal. In other words, we have this alternative implementation of the typical GP:

where now the parameter vector. This is equivalent to the definition above.

Note that, even though the expected prior mean of is zero, its posterior mean (after fitting it to data) is not necessarily zero.

Mean-zero Gaussian processes

We define a mean-zero GP starting from the Cholesky decomposition implementation

First we place a sum-to-zero constraint on :

We then define

This is the mean-zero GP.

Applications in the MMM

We use these mean-zero GPs to build the MMM’s time-varying parameters.

Suppose we want to build a channel’s beta parameter, . First we build a vector on unconstrained space by mixing the mean-zero GP with a scalar parameter :

where

• is a scalar between 0 and 1,

• is a scalar with ,

• and is a mean-zero GP. We scale it so that each entry of has a prior variance of 1.

We construct the final beta parameter using the inverse logit transform:

where , , and are constants we can use to set appropriate priors on .

We use this construction for essentially every time-varying parameter in the MMM.

Why?

Constructing like this has some nice benefits.

1. The scalar acts as a hyperparameter that controls how much of comes from or from . If is small then is approximately constant, and if is large then is approximately a mean-zero GP.

2. Putting and in square roots makes it so that no matter what is, each entry of always has a prior variance of approximately 1.

3. We have , giving us a simple parameter we can use to share information across channels. Indeed, when we link channels in the model, we are allowing their parameters to influence one another.

Appendix: Motivation for the construction

The original motivation for mean-zero GPs came from wanting a single parameter that acted as the mean value of the GP.

Naively we tried taking a typical GP and a scalar and defining a time-varying parameter like

but ran into a major issue. It turns out that when you use the Cholesky decomposition method to build the for this, becomes nonidentifiable.

If , then define alternate versions of and -- we’ll call them and -- by

where is any scalar. Then

and so

Since we’ve found infinitely many different values of (since was arbitrary) that produce the same .

You might be interested to see what that vector looks like. Here’s a plot using one of our usual kernels:

What can we do to constrain so that it can’t move along this problem vector? Well, since this problem vector changes the sum of , we could try restricting so that its sum is zero.

If we force to sum to zero, we effectively restrict our GP to a subset of all typical GPs. How much does this constrain ? Not much, it turns out. We can easily generate some data to try fitting it with these constrained GPs, and in doing so we found that the constrained GPs behave quite similarly to unconstrained GPs.

The final step, going from

to

gives us a new hyperparameter that controls how constant-like is. We could achieve a similar result by changing the variance hyperparameter in the GP kernel, but by doing it this way we don’t need to worry about balancing the prior variances of and .

Footnotes

[1] A great intro to Gaussian processes is Yuge Shi’s “Gaussian Processes, not quite for dummies”, The Gradient, 2019.