The Forget-me-not Process
We introduce the Forget-me-not Process, an efficient, non-parametric metaalgorithm for online probabilistic sequence prediction for piecewise stationary, repeating sources. Our method works by taking a Bayesian approach to partitioning a stream of data into postulated task-specific segments, while simultaneously building a model for each task. We provide regret guarantees with respect to piecewise stationary data sources under the logarithmic loss, and validate the method empirically across a range of sequence prediction and task identification problems.
The authors propose a scheme suitable for the predictions of piecewise "stationary" sequences, where here "stationarity" merely means that the sub-sequence corresponding to each "stationary" component is efficiently learnable by a given prediction scheme. The main challenge in this setting is therefore efficient identification of the boundaries between these components. The paper builds upon Bayesian mixing schemes and particularly PTW, and its novelty is in being able to exploit repetition within the components. The authors present an algorithm, along with several practically crucial complexity reduction heuristics, as well as several experiments studying the efficacy of their approach.
Overall, I believe this is a good paper - the algorithm appears non-trivial and innovative, and the experiments are convincing. However, the paper also has, in my opinion, several shortcomings:  1) There were several technical aspects of the algorithm and its implementation that I could not understand: A) In general, have found the pseudocode  for the algorithm hard to follow, despite reading the entire paper; I would recommend more verbal description (perhaps as comments next to the pseudocode) and if possible, less notation. B) The combination of the method with MADE is not clear to me. In particular, when and for which models gradient steps are taken? Are mini-batches used? Does the AdaGrad optimizer keep a separate state for each model? While I understand that the details of the training of the prediction models can be abstracted away while considering the mixing scheme, I think presenting these details when describing the algorithm is important and can make the presentation far more accessible. C) Two aspects of the complexity reducing heuristics were not clear to me: I) If rho* is selected as the best performing model for the segment, it seems unlikely for the condition of option 2 of the pruning heuristic to ever happen. In particular, if no sub-sampling is used, shouldn't rho*(s) be greater than xi(s) by construction? II) In option 3 of the pruning heuristic, how exactly is the "age" of a model determined?   D) How were the hyper-parameters determined in each experiment?  2) The efficacy of the approach in exploiting repetition is not established theoretically. Instead, the guarantees of PTW are shown to be inherited, and efficacy for repetition is deferred to experimental studies. Moreover, the theory doesn't seem to consider the complexity reducing heuristics, without which the approach has very limited practical value. I believe that the "Theoretical Properties" paragraph of the paper is sufficiently weak to be relegated to supplementary materials, and the freed up space can be used for more explanations of other aspects of the paper.  3) Motivation for the work is lacking in the sense that no convincing applications are described. The only "real-world" application for the proposed algorithm mentioned in the paper is multiple-task reinforcement learning, with a stream of different Atari games given as an example. However, it is not clear to me why should the identity of the game played be hidden from the agent learning multiple Atari games. Note that while I feel motivation is lacking in the presentation, I do believe the suitable applications exist, and answered question 7 of the review accordingly. I believe the authors should provide additional motivating examples in the introduction to the paper.  Here are some additional comments and suggestions: a) The abstract refers to the innovative aspect of this work with only one word ("repeating"). I believe this should be expanded to at least one sentence (I only understood that this is the innovative aspect after reading the introduction). b) In the denominator just below line 86, I believe it should be rho instead of nu c) In line 129, I believe it should be rho[x_{a_{i-1}:b_{i-1}}] rather than rho[x_{a_{i}:b_{i}}] d) In line 166, is "marginal probability" the right term here? Shouldn't it be "joint probability"? e) In line 177, the assumption that prediction takes O(1) time is perhaps not detailed enough when considering neural networks as statistical models; instead, it should be interesting to count exactly the number of forward and backward passes, as well as parameter updates, that the algorithm uses. f) The use of a generative model such as MADE in an online fashion is, to my knowledge, quite non-standard. It should be interesting to see a graph of the commutative regret as a function of time in order to get more insight into the learning dynamics and convergence behavior of the model. g) In line 259, the oracle is reported for the FOD domain, not the MBOC.