Decisions made by machine learning systems have increasing influence on the world. Yet it is common for machine learning algorithms to assume that no such influence exists. An example is the use of the i.i.d. assumption in online learning for applications such as content recommendation, where the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users. Generally speaking, it is possible for an algorithm to change the distribution of its own inputs. We introduce the term self-induced distributional shift (SIDS) to describe this phenomenon. A large body of work in reinforcement learning and causal machine learning aims to deal with distributional shift caused by deploying learning systems previously trained offline. Our goal is similar, but distinct: we point out that changes to the learning algorithm, such as the introduction of meta-learning, can reveal hidden incentives for distributional shift (HIDS), and aim to diagnose and prevent problems associated with hidden incentives. We design a simple environment as a 'unit test' for HIDS, as well as a content recommendation environment which allows us to disentangle different types of SIDS. We demonstrate the potential for HIDS to cause unexpected or undesirable behavior in these environments, and propose and test a mitigation strategy.
The authors study the phenomena of self-introduced distributional shift. They define the term along with the term hidden incentives for distributional shift. The latter describes factors that motivate the learner to change the distribution in order to achieve a higher performance. The authors study both phenomena in two domains (one being a prisoner dilemma and the other a recommender system) and show how meta-learning reveals the hidden incentives for distributional shift. They then propose an approach based on swapping learners between environments to reduce self introduced distributional shift.
In my opinion this paper should be (weak) rejected for several reasons.
-	The paper is poorly written. Some sentences are hard to comprehend, even after repeated reading. It feels hastily written and should be carefully proofread with the help of a proficient English speaker.
-	The very first paragraph is not a helpful example. While the authors provide an example that that illustrates a distributional shift, they don’t describe what the shift is and why that shift in a negative consequence for learning. Therefore, the example is not helpful, but rather confusing.
-	In point 3 of the contributions, the authors state ‘that meta-learning reveals HIDS in these environments’. Is this true for all meta-learning approaches? The current statement overclaims their findings. If the authors decide to keep it, they should provide a description of the types of meta-learning approaches that reveal HIDS and which don’t or evaluate all different meta-learning approaches.
-	Section 4.1 is hard to follow. The authors state that the meta-learner is used to tune the learning rate – they fail to clearly explain how exactly the learning rate tuning results in a non-myopic behavior for a myopic learner.
-	In the Q-learning example in Section 4.1, is the effect of \epsilon considered? The discovery of non-myopic strategies might simply be based on chance. I would like the authors to include an investigation into the effect of this parameter.
Generally speaking, I appreciate the author’s thoughts on SIDS and how they approach revealing hidden incentives. Although I vote to reject this paper, I strongly encourage the authors to rewrite the paper, address all other issues that are noted during this peer review and resubmit.