Exploiting easy data in online optimization
We consider the problem of online optimization, where a learner chooses a decision from a given decision set and suffers some loss associated with the decision and the state of the environment. The learner's objective is to minimize its cumulative regret against the best fixed decision in hindsight. Over the past few decades numerous variants have been considered, with many algorithms designed to achieve sub-linear regret in the worst case. However, this level of robustness comes at a cost. Proposed algorithms are often over-conservative, failing to adapt to the actual complexity of the loss sequence which is often far from the worst case. In this paper we introduce a general algorithm that, provided with a "safe" learning algorithm and an opportunistic "benchmark", can effectively combine good worst-case guarantees with much improved performance on "easy" data. We derive general theoretical bounds on the regret of the proposed algorithm and discuss its implementation in a wide range of applications, notably in the problem of learning with shifting experts (a recent COLT open problem). Finally, we provide numerical simulations in the setting of prediction with expert advice with comparisons to the state of the art.
This work presents a new way to exploit 'easy' data in online convexoptimization and in the bandit setting. Given a strategy A that has goodworst-case guarantees, and another opportunistic strategy B that might,for some data, get much smaller regret, it is shown how to combine A andB into a single strategy that gets the best of both worlds, at the costof adding O(sqrt(T*log(T))) to the regret of A and a small constant tothe regret of B. The same approach is shown to work for prediction withexpert advice, online convex optimization and a relaxation of the banditsetting in which *two* arms can be observed in every round.This is pretty cool, because previous authors have needed to usecomplicated special-purpose strategies that depended on A and B toachieve similar results.Applications are given that 1) take B to be Follow-the-Leader and getperformance similar to the FlipFlop algorithm of De Rooij et al. [8]; 2)address a recent open problem about tracking the best expert, and 3)adapt to worst-case and stochastic bandits as in the work of Bubeck andSlivkins [5]. In these applications, the results of prior work are notexactly recovered, but the authors generally provide a good discussionof the differences.The paper makes me wonder about a natural follow-up question: Is itpossible to generalize the approach to include more than oneopportunistic benchmark strategy B and get small regret with respect toall of them? Currently the additive term O(sqrt(T*log(T))) preventsnesting the approach multiple levels.Some remarks:Tracking the best expert:  - the proposed method needs to know the gap delta between the best    expert in each interval and the second-best, which is unknown in    practice. I would suggest adding a comment that mentions this.The comparison to FlipFlop on p.5:  - This comparison focusses on using specific parameters for FlipFlop.    These parameters trade off the performance in the worst-case with    the overhead with respect to FTL. So perhaps a fairer comparison    could be made by setting them to make the worst-case for FlipFlop    and for (A,B)-Prod the same. Alternatively, the discussion might be    simplified by not focussing on particular parameter values too much.  - The comment about competing with the actual regret of    AdaHedge vs its regret bound seems misplaced, because the difference    between the two is negligible compared to the additional    sqrt(T*log(T)) term. line 267: why does D-Prod guarantee regret of O(T^{1/4}) when FTL and    AdaHedge both have regret Theta(sqrt(T))? Suppose we generate losses    for 2 experts from the uniform distribution (as in the lower bound    from "How to use expert advice"), then any algorithm will have    regret Theta(sqrt{T}), so how can this be true? Which algorithm is    used as the benchmark in this case?Minor comments:In figure 2, w_{1,A} is not really a parameter, because it is fixed to  be 1-w_{1,B}.In Thm 1, why introduce mu_A and mu_B, when they are the same as  w_{1,A} and w_{1,B}?In Corollary 1, I think your result requires C >= e^4 or something.line 245: "a factor of" -> "an additive term of" (factors are multiplicative)Proposition 1: say that there is a proof in the appendixIn the discussion of your result for the bandit setting, you could also  mention the recent paper "One Practical Algorithm for Both Stochastic  and Adversarial Bandits" by Seldin and Slivkins at ICML 2014.line 488: there is a minus missing: -log(1-eta)/eta is increasingline 497: doesn't this hold with equality?