How To Set Up A Holdout Group For Advertising Tests

Holdout Group A group intentionally excluded from advertising so its outcomes can be compared with exposed users or markets. Setting up a holdout correctly requires deliberate choices about unit of randomization, sample size, exposure control, measurement windows, and monitoring to prevent contamination.

Begin with the objective: are you measuring short-term conversions, long-run lifetime value, or incremental retention? That decision drives the length of the measurement window and the size of the holdout. For direct response campaigns you might need only a few weeks; for subscriptions or durable goods you may need several months to capture delayed effects.

Step-By-Step Setup

• Define The Unit Of Randomization: Choose individuals, households, accounts, or markets. Individual-level randomization yields higher statistical power but may be impractical for mass media.
• Calculate Sample Size And MDE: Use baseline conversion rates, desired statistical power (usually 80% or 90%), and the smallest effect size you care about to compute required sample sizes for both exposed and holdout groups.
• Randomize And Stratify: Random assignment is essential. Stratify on major covariates (region, device type, prior spend) to ensure balanced groups when those covariates affect outcomes.
• Implement Suppression: Enforce exclusion across ad platforms and partners. Build suppression lists, apply DSP/SSP blocking, and coordinate with media partners for linear channels where possible.
• Set Measurement Window: Specify pre-period for balance checks and post-period long enough to capture conversion behavior relevant to the business.

Operational Details And Platform Considerations

Ad platforms often identify audiences using cookies, mobile advertising IDs, or hashed identifiers. Consistency matters: use the same identity mapping for both suppression and outcome measurement. Cross-device issues require household or deterministic identifiers when possible; otherwise expect some contamination. For offline channels like TV, prefer geographic holdouts that remove entire DMAs from buys.

Monitoring And Validation

Run pre-period balance checks on key metrics to confirm comparability. During the test, monitor suppression rates, ad exposures among holdout segments (to detect leaks), and baseline trends. After the test, compute incremental lift with confidence intervals and run sensitivity checks such as excluding edge-case cohorts or running difference-in-differences if trends diverged pre-test.

List Of Practical Tips

• Plan For Contamination: Expect some leakage; estimate its size and report conservative adjusted lift figures if necessary.
• Coordinate Cross-Functional Teams: Involve media buyers, analytics, legal, and engineers early to ensure suppression and measurement align.
• Document Everything: Track assignment algorithms, suppression logic, and versioning of creatives so post-mortem analysis is reproducible.
• Use Automated Tooling: Where possible, leverage experimentation platforms or CDPs that support holdout management and measurement pipelines.

In short, the Holdout Group is a core element of credible advertising measurement. Define your unit, power the test with adequate sample size, enforce strict exposure control, choose the right measurement window, and monitor for contamination to turn a simple exclusion into reliable incremental insight.