Near Optimal AB Testing

Abstract

We consider the problem of A-B testing when the impact of the treatment is marred by a large number of covariates. Randomization can be highly inecient in such settings, and thus we consider the problem of optimally allocating test subjects to either treatment with a view to maximizing the precision of our estimate of the treatment eect. Our main contribution is a tractable algorithm for this problem in the online setting, where subjects arrive, and must be assigned, sequentially, with covariates drawn from an elliptical distribution with finite second moment. We further characterize the gain in precision aorded by optimized allocations relative to randomized allocations, and show that this gain grows large as the number of covariates grow. Our dynamic optimization framework admits a number of generalizations that incorporate important operational constraints such as the consideration of selection bias, budgets on allocations, and endogenous stopping times. In a set of numerical experiments, we demonstrate that our method simultaneously oers better statistical eciency and less selection bias than state-of-the-art competing biased coin designs.

Publication
In Management Science
Date