Feedback Detection for Live Predictors
A predictor that is deployed in a live production system may perturb the features it uses to make predictions. Such a feedback loop can occur, for example, when a model that predicts a certain type of behavior ends up causing the behavior it predicts, thus creating a self-fulfilling prophecy. In this paper we analyze predictor feedback detection as a causal inference problem, and introduce a local randomization scheme that can be used to detect non-linear feedback in real-world problems. We conduct a pilot study for our proposed methodology using a predictive system currently deployed as a part of a search engine.
The authors study situations in which predictions influence future inputs ofthe predictor, thereby introducing feedback into the system. The authors assumethat the predictor is linear in terms of its inputs, and model the feedback asa term that is added to the new prediction at $t+1$ and may depend nonlinearlyon the old prediction at $t$. The authors show that under these assumptions, itis possible to estimate the feedback function by adding independent noise tothe prediction. The method is similar in spirit to the "randomization" used inrandomized clinical trials, or in other situations in which the goal is toidentify a causal effect (instead of merely estimating statisticalassociation). The independence of the added prediction noise allows one toestimate the desired causal effect (which is the feedback, in this case).Although it is always difficult with articles like these to judge the novelty(as similar ideas might have come up in very different disciplines, rangingfrom econometry, engineering, control theory to causality as a subdiscipline ofAI and/or statistics), I haven't seen similar ideas before in the context oflearning systems. The most relevant related work that addresses feedback loops isall the work about "position bias" in the learning-to-rank literature. A common way to address feedback loops is to use interleaved comparison methods. This exploits the factthat if the prediction is a ranking, one can compare two different rankers by only outputting a single ranking (which combines, in an interleaved way, the rankings obtained by the two rankers that are compared), thereby allowing to partially observe factual and counterfactuals simultaneously.Nevertheless, the problem setting is interesting and the way in which it is formalized seems to be original. The analysis by the authors seems technically correct (although I didn't check the Theorems in detail) and I enjoyed reading the article. As I think the problem of feedback in live predictor systems is an interesting one that may becomemore relevant in the future (as machine learning gets more widely deployed) andmay deserve more attention in the NIPS community, I recommend acceptance.Two aspects with respect to which the article can still be improved are:(1) convincing the reader more of the significance of the problem that is addressed;(2) being more explicit about the assumptions that are made. For example, it would be very instructive to show how one could think about the additive-feedback assumption as a firstorder approximation (by some Taylor expansion argument). Also, what assumptions are made about the "natural process" $x^{(t)}$? It seems to me that the authors assume some kind of Markov property. In lines 211-214, the authors make the assumption that the predictor is linear. Is that assumption really necessary, or is the only relevant starting point for their analysis the equation in line 212-213 (basically, the additive feedback equation), which could possibly be interpreted as a first-order approximation of some more complicated nonlinear system (thereby allowing for nonlinear predictors)?Some small comments:o Why call this "statistical feedback"? Wouldn't something like "predictorfeedback" be a more descriptive term?o Lines 44-45: "This paper proposes a fully automatic method for detectingstatistical feedback loops that does not require modeling the feedbackmechanism." This sentence is misleading, as the authors DO model the feedbackmechanism (as an additive term that may depend nonlinearly on the lastprediction, see line 211). Another assumption about the feedback process thatthe authors make is that the prediction $\hat y^t$ only directly influences$x^{t+1}$ (it could also directly influence $x^{t+i}$ for any $i > 1$). Itherefore suggest reformulating or weakening this claim.o Line 47: "If future model predictions increase when we add artificial noiseto the current ones, then our system has a feedback problem." This sentence isconfusing. It would help to replace the word "increase" by "change" (becausethe sign of the change depends on the noise value - even assuming a positivecorrelation between noise and feedback, then the future model predictions onlyincrease when the noise is positive, and they decrease when the noise isnegative). Also, it is not clear why the feedback is a "problem". As shown inthe pilot study, feedback is not necessarily a problem (although it may becomeone if the feedback is too strong).o Line 211: what do you mean by "feedback function in $x$-space"?o Line 213: shouldn't this be $f(\cdot) = w \cdot f_{(x)}(\cdot)$?o (10): for consistency, use $\phi_N$ instead of $\phi_{\sigma_\nu}$