Experimentation

THE foundational paper on web experimentation. Covers hypothesis testing, sample size, metrics, and common pitfalls. 2000+ citations.

Google's infrastructure for running overlapping experiments, enabling hundreds of simultaneous tests without interference.

The definitive SRM paper from Microsoft/Booking.com—provides taxonomy of causes and 10 diagnostic rules; ~6% of experiments exhibit SRM.

Foundational Microsoft paper on experiment validity; coined OEC design principles and trustworthiness checks now industry standard.

Essential guide to metric design—introduces metric taxonomy (OEC, guardrail, diagnostic) with real experiment examples.

The CUPED method that uses pre-experiment covariates to dramatically reduce variance in experiment metrics.

Shows how regression adjustment in randomized experiments yields valid inference even with heterogeneous treatment effects.

Optimal stratification and regression adjustment methods for experimental data with theoretical guarantees.

Introduces MLRATE—extends CUPED to ML models via cross-fitting; achieved 70%+ variance reduction over difference-in-means at Meta.

Theoretical foundation for post-stratification; proves it's nearly as efficient as blocking with variance difference O(1/n²).

Industry workhorse comparing stratification, post-stratification, and CUPED at scale; recommends post-assignment techniques.

Analyzes the peeking problem in A/B tests and introduces always-valid p-values for sequential testing.

E-values and safe anytime-valid inference that allows optional stopping while controlling type I error.

Addresses the problem of highly variable metrics in experiments and proposes robust statistical methods.

Foundational classic (3,200+ citations)—established group sequential boundaries that all modern methods build upon.

Modern theoretical foundation for anytime-valid inference; backbone for GrowthBook, Spotify, and other industry tools.

Rigorous theoretical grounding for Bayesian sequential testing with continuous monitoring; Microsoft's framework.

Graph cluster randomization and design-based methods for detecting interference in network experiments.

Two-stage randomization design for identifying peer effects in social networks.

Framework for experiments in marketplaces where treatment of one side affects the other.

The seminal exposure mapping paper—unified framework for design, exposure mapping, and estimands under network interference.

Essential for two-sided marketplaces—introduces ERL estimator for buyer/seller and rider/driver experiment structures.

Comprehensive review of spatial interference methods including geostatistical approaches for location-based spillovers.

Design and analysis of switchback experiments for marketplace interventions at Airbnb.

Google's Bayesian structural time-series approach for measuring impact of geo-level interventions.

Meta's open-source tool for geo-experiment design and measurement using synthetic control methods.

The definitive switchback paper—optimal design under carryover effects with minimax formulation; used by Uber, Lyft, DoorDash.

Authoritative methodological review from synthetic control's creator; called 'most important innovation in policy evaluation in 15 years'.

Addresses power analysis for geo experiments with few heterogeneous regions; foundation for Google's Trimmed Match library.

Framework for using short-term outcomes to predict long-term treatment effects in experiments.

Methods for combining experimental and observational data to estimate persistent treatment effects.

First scalable method to estimate user-learning effects (novelty/primacy) via difference-in-differences across thousands of Microsoft experiments.

Largest empirical validation of surrogates—95% consistency between 14-day surrogate predictions and 63-day outcomes across 1,098 test arms.

First rigorous method for high-dimensional surrogates where number of surrogates exceeds sample size; essential for multi-metric settings.