A framework for Multi-A(rmed)/B(andit) Testing with Online FDR Control
We propose an alternative framework to existing setups for controlling false alarms when multiple A/B tests are run over time. This setup arises in many practical applications, e.g. when pharmaceutical companies test new treatment options against control pills for different diseases, or when internet companies test their default webpages versus various alternatives over time. Our framework proposes to replace a sequence of A/B tests by a sequence of best-arm MAB instances, which can be continuously monitored by the data scientist. When interleaving the MAB tests with an online false discovery rate (FDR) algorithm, we can obtain the best of both worlds: low sample complexity and any time online FDR control. Our main contributions are: (i) to propose reasonable definitions of a null hypothesis for MAB instances; (ii) to demonstrate how one can derive an always-valid sequential p-value that allows continuous monitoring of each MAB test; and (iii) to show that using rejection thresholds of online-FDR algorithms as the confidence levels for the MAB algorithms results in both sample-optimality, high power and low FDR at any point in time. We run extensive simulations to verify our claims, and also report results on real data collected from the New Yorker Cartoon Caption contest.
The paper looks at continuous improvement using a sequence of A/B tests, and proposes instead to implement adaptive testing such as 
      multi-armed bandit problem while controlling the false discovery rate. This is an important problem discussed in statistical literature, but still unsolved.
      The approach proposed in this paper seems to apparently solve the issues. This is a very interesting paper, that despite minor concerns listed below, could lead to a potentially new avenue of research.
      
      Lines 24-37: There are well-known issues, and it would be desirable to add citations. Although authors clearly focus on CS/ML literature, there is also a relevant body of literature in biometrics, see e.g. survey by Villar, Bowden and Wason (Statistical Science 2015), the references therein and the more recent papers citing this survey.
      Line 37: "testing multiple literature" -> "multiple testing literature" 
      Line 38-47: A similar concept exists in biometrics, called "platform trials" - please describe how your concept differs
      Line 112: "and and" -> "and"
      Line 115: please provide reference for and description of LUCB
      Line 153: "samplesas" -> "samples as"
      Line 260: "are ran" -> ?
      Line 273: It is not clear what "truncation time" is and why it is introduced - it seems to have a huge effect on the results in Figure 2
      Line 288-291: While this motivation is interesting, it seems to be mentioned at an inappropriate place in the paper - why not to do it in the introduction, alongside website management and clinical trials?