Launch and Iterate: Reducing Prediction Churn
Practical applications of machine learning often involve successive training iterations with changes to features and training examples. Ideally, changes in the output of any new model should only be improvements (wins) over the previous iteration, but in practice the predictions may change neutrally for many examples, resulting in extra net-zero wins and losses, referred to as unnecessary churn. These changes in the predictions are problematic for usability for some applications, and make it harder and more expensive to measure if a change is statistically significant positive. In this paper, we formulate the problem and present a stabilization operator to regularize a classifier towards a previous classifier. We use a Markov chain Monte Carlo stabilization operator to produce a model with more consistent predictions without adversely affecting accuracy. We investigate the properties of the proposal with theoretical analysis. Experiments on benchmark datasets for different classification algorithms demonstrate the method and the resulting reduction in churn.
The paper articulates an important problem---reducing unnecessary changes in the predictions of machine learning algorithms, describes why this problem is relevant (it hinders usability of prediction, and hinders ability to statistically validate that the trained model improved), and gives this problem a name (“prediction churn”) and very general formal definition.  The paper introduces a conceptually interesting and general fixed-point approach to reducing prediction churn, which it describes (in a somewhat hand-wavy way) as an MCMC technique. The approach combines regularization of new classifiers with respect to the previous classifier and training the new classifier on perturbed version of the training data set. Together, these techniques, when iterated, are expected to result in a final classifier that will not change much when re-trained (regularized) on the perturbed dataset when the perturbations are the result of real-world drift. The paper provides examples of two fairly general stability operators with hyperparameters governing the tradeoff between churn and accuracy.  The paper provides some theoretical bounds on the “churn” for restricted settings, but does not provide theoretical characterization of the de-churning of the iterative Markov chain, such as when a stationary distribution exists.  The paper includes experiments on real-world datasets and several learning algorithms that demonstrate that a one-step and Markov chain de-churning process successfully reduce churn while maintaining or improving accuracy. The experimental results do not emphasize the importance of the fixed-point property of the Markov chain.
The problem articulated and addressed by the paper is important, the techniques introduces are very general in their applicability, and the experimental results successfully demonstrate the utility of the techniques to addressing the problem.  The fixed-point MCMC approach is very conceptually compelling, although it lacked theoretical analysis and rigor, and appeared to be not that significant to achieving good results (the results suggest a single iteration may be sufficient). After reading the highly suggestive section on the Markov chain, the theoretical results appeared underwhelming. A rigorous analysis of the Markov chain would have made the paper more self-contained and increased its technical strength. Alternatively, focusing on the single-step stabilization operator case, and obtaining more general theoretical results within that restricted setting, would have made the paper more self-contained and cohesive.  The paper lacked polish, which made it harder to digest than necessary. For example, in the experiment section 5.1 the authors paper refers to "the RCP operator comparison" and "the MCMC operator comparison", and do not make it clear that the MCMC comparison itself is composed of multiple RCP operators. Issues of wording like this, as well as somewhat strange choices of order in which new notation is presented (e.g. T_A is presented on line 87 even though it is not necessary until later; the use of undefined notation P_T on line 91), reduced the clarity of the paper.  It would have been interesting to see a relation of the technique to online Bayesian learning. The Markov chain, stabilization operator, and training data perturbation technique seem related to Bayesian approaches, and an analysis of the Markov chain (that was not included in the paper) may specifically benefit from this perspective.  Finally, the paper would have benefited from a discussion or conclusion that discussed in more detail the potential impact of this work.