  Motivated by the widespread adoption of large-scale A/B testing in industry,
we propose a new experimentation framework for the setting where potential
experiments are abundant (i.e., many hypotheses are available to test), and
observations are costly; we refer to this as the experiment-rich regime. Such
scenarios require the experimenter to internalize the opportunity cost of
assigning a sample to a particular experiment. We fully characterize the
optimal policy and give an algorithm to compute it. Furthermore, we develop a
simple heuristic that also provides intuition for the optimal policy. We use
simulations based on real data to compare both the optimal algorithm and the
heuristic to other natural alternative experimental design frameworks. In
particular, we discuss the paradox of power: high-powered classical tests can
lead to highly inefficient sampling in the experiment-rich regime.

Machine Learning;Machine Learning