  Randomized experiments play a major role in data-driven decision making
across many different fields and disciplines. In medicine, for example,
randomized controlled trials (RCTs) are the backbone of clinical trial
methodology for testing the efficacy of new drugs and therapies versus existing
treatments or placebo. In business and marketing, randomized experiments are
typically referred to as A/B tests when there are only two arms, or variants,
in the experiment, and as multivariate A/B tests when there are more than two
arms. Typical applications of A/B tests include comparing the effectiveness of
different ad campaigns, evaluating how people respond to different website
layouts, or comparing different customer subpopulations to each other.
  This paper focuses on multivariate A/B testing from a digital marketing
perspective, and presents a method for the sequential monitoring of such
experiments while accounting for the issue of multiple comparisons. In adapting
and combining the methods of two previous works, the method presented herein is
straightforward to implement using standard statistical software and performs
quite well in various simulation studies, exhibiting better power and smaller
average sample sizes than comparable methods.

Applications;Methodology