Holdout Sample and Definition

[SteliosSake]

Hi everybody,

I would like to ask the following questions regarding the holdout sample when fitting a model:

• I have seen from discussions and documentation that, since meridian's task is causal inference (and not prediction), we would prefer some overfitting in some cases (the results causally make sense and align with reality but the MAPE is slightly worse). Hence leading to the question: Is it "mandatory" in terms of validity to utilize the holdout-id argument and have a holdout sample during fitting? (because it will surely lead to a better model) Or is it preferred in some cases to not have any holdout sample and fit the model only on training data?

• The definition of the holdout sample must be appropriate, meaning that if many knots are utilized, data points around the knots should not be part of the holdout sample since it will lead to the knot being calculated by the prior distribution (this can lead to another problem of a negative R^2). Is there a particular rule of thumb regarding the definition of the holdout sample around the knots? (e.g., there should be at least a 2 data point gap between holdout data points if they are close to a knot or knot data points should not be in the holdout sample)

Thanks in advance for all your help!

Kind regards,
SteliosSake

[cpulavarthi]

Hello @SteliosSake,

Thank you for reaching out to us with detailed questions!

• You are correct about Meridian’s primary purpose being modeling causal relationships between media efforts and the target KPI. In terms of the implementation, the holdout_id isn’t a mandatory argument for modeling. Defining the holdout set helps in evaluating the model performance by computing out-of-sample predictive accuracy metrics. This is not only useful in understanding the generalizability of the model (i.e., knowing whether the model overfits), but also acts as a diagnostic tool and is useful while comparing multiple models. We recommend holding out some observations to better understand the model performance. If you have limited data where every observation is critical for parameter estimation, you could choose to forgo a holdout. Or you may still build 2 models, one with a holdout set and the other without, and compare them.
• For Meridian, we recommend using a holdout sample which is fairly balanced across geos and time periods. For this, it is better to use a random sample rather than strategizing holdout points around knot locations. If you end up choosing a holdout set with a pattern across time, like one datapoint held out for every month at the same time, then you may introduce bias because the true trend for that particular period of the month may not be captured, hence the recommendation to use random holdout samples.
  • Your understanding about why holdout and knot locations must not coincide is correct, i.e., this may lead to knot posteriors not being estimated properly due to lack of data. To avoid holding out data from knot locations, you may choose to randomly select the holdout set after selecting the knot placements.
  • If you are fitting a geo model, it is recommended to create holdout ids by randomly selecting x% geos per time. This will guarantee that every time period has some observations used for training and some used for prediction, and thus knot placement won't be a concern any more.

Feel free to go through our documentation on Holdout Sets for more information regarding this.

Do contact us if you have any further queries or suggestions regarding Meridian.

Google Meridian Support Team

[SteliosSake]

Hello @cpulavarthi ,

Thank you for your quick response! The input really helped a lot.

Regarding the points you raised, I have the below follow up questions:

"Your understanding about why holdout and knot locations must not coincide is correct, i.e., this may lead to knot posteriors not being estimated properly due to lack of data. To avoid holding out data from knot locations, you may choose to randomly select the holdout set after selecting the knot placements."

• So the best practice is to define my knot locations and then just randomly sample from the set of data points that are NOT knot locations. Did I understand that correctly?
• The only strategy that I am utilizing right now regarding the holdout definition is 1.) that we don't want a holdout to be at a knot location and 2.) that we don't want to have more that two consecutive holdout sample data points. The holdout is randomly selected from the set of data points that is not a knot location and afterwards a check happens to see if there are more the two consecutive data point selection, which are then corrected. Is this a method that could introduce bias?

"This is not only useful in understanding the generalizability of the model (i.e., knowing whether the model overfits), but also acts as a diagnostic tool and is useful while comparing multiple models."

• Since the holdout sample is randomly selected, it leads to the goodness of fit metrics to have a certain variance (for different holdout sample locations we get different goodness of fit R^2 and MAPE, wMAPE). When dealing with a scenario like this, my intuition leads me to run the fitting of a single model n number of times (e.g., 5-10 times) for different randomly selected holdout samples, and then taking the mean of the GOF metrics. If I want to compare two models with each other, then I will do the previous strategy for both models and then compare their means of GOF metrics. Do you believe that this is a good practice to utilize when trying to validate the fitting of your model with a holdout sample and also for comparing 2 models with each other?

"If you have limited data where every observation is critical for parameter estimation, you could choose to forgo a holdout"

• I am utilizing a national model that has 125 weeks of data (2.4 years of data). The spend treatments /marketing channels are 12. In your opinion are those data considered limited?

Thanks in advance for all your help and time! Let me know if you need any more information from my side.

Kind regards,
SteliosSake

[cpulavarthi]

Hello @SteliosSake,

Thank you again for reaching out and including all relevant information needed to better help you with your queries. I shall answer them in the same order.

1. • Yes, I would recommend defining knots first and then selecting a random sample from the remaining data points.
   • The strategy you described has one additional step, i.e., manually removing any holdout points that were consecutive. This won't be introducing any bias. The reason I didn't mention this is because the probability of choosing consecutive points will be low in a 15% - 20% holdout selection, and consecutive holdout points won't be affecting model training too much either because the model still uses data from the treatment variables for training (as it would be needed to compute carryover effects), it only avoids using the dependent variable (the target KPI values) for those held out periods. So the additional work feels unnecessary, we only recommend avoiding holding out large, contiguous-in-time, chunks of data, anything beyond that will be a personal preference.

2. The strategy you described is known as repeated random sub-sampling validation, which is a robust form of cross-validation. It is definitely a good practice and could help guide your decisions better. However, fitting a model 5-10 times may be time consuming and requires significant computational resources, and as long as this isn't a constraint, you can continue with this approach.

3. The data requirements for Meridian are detailed in our documentation linked here.

   "As a general rule of thumb, historical data should be a minimum of two years' worth of weekly data for geo-level models and three years' of data for national-level models."

   While the dataset you described has lesser amount of data than ideal, you may choose to combine some of the channels into a single one if having less data is causing convergence issues. The amount of data required is subjective, it depends on the number of model parameters to be estimated, quality of the dataset itself, etc. Sound modeling choices (such as setting appropriate priors, strategically choosing knot locations, etc.) can help you get better results even with limited data.

Hope this addresses all your concerns. Do contact us if you have any further questions regarding this.

Google Meridian Support Team

[amalinsky]

Hello, I'd like some details on how I could isolate holdout samples in Model Diagnostics.
Currently I declared "holdout_id" at model building and trained a model.
But it's not mirrored in model diagnostics.

I could still join both holdout array and data to figure out the prediction accuracy on those particular datapoints though, do you confirm ?

[gousesb]

Hello @amalinsky ,

Thank you for contacting us!

The primary purpose of the holdout_id in Meridian's Model Diagnostics is not for forecasting, but to serve as a confidence check by measuring out-of-sample predictive accuracy. This allows you to evaluate if the model structure is appropriate and not overparameterized, and is crucial for comparing different model specifications like prior strengths. Since the holdout sample is intended for calculating these metrics, Meridian's built-in tools can retrieve them directly by instantiating the visualizer.ModelDiagnostics object and calling the predictive_accuracy_table(), you will get the R^2, MAPE, and wMAPE for the Test (Holdout) subset, provided the holdout_id was set during modeling. See here for more information.

model_diagnostics = visualizer.ModelDiagnostics(meridian)
model_diagnostics.predictive_accuracy_table()

However, you method is approachable if you need the row-level predictions for custom analysis or residual checks, you must manually join your original holdout_id array (with the unique identifiers) to the model's prediction output data (containing the actual and predicted values) to isolate those specific data points.

Feel free to reach out for any further queries or feedback regarding Meridian.

Google Meridian Support Team