Hidden incentives for self-induced distributional shift
Decisions made by machine learning systems have increasing influence on the world. Yet it is common for machine learning algorithms to assume that no such influence exists. An example is the use of the i.i.d. assumption in online learning for applications such as content recommendation, where the (choice of) content displayed can change users* perceptions and preferences, or even drive them away, causing a shift in the distribution of users. Generally speaking, it is possible for an algorithm to change the distribution of its own inputs. We introduce the term self-induced distributional shift (SIDS) to describe this phenomenon. A large body of work in reinforcement learning and causal machine learning aims to deal with distributional shift caused by deploying learning systems previously trained offline. Our goal is similar, but distinct: we point out that changes to the learning algorithm, such as the introduction of meta-learning, can reveal hidden incentives for distributional shift (HIDS), and aim to diagnose and prevent problems associated with hidden incentives. We design a simple environment as a *unit test* for HIDS, as well as a content recommendation environment which allows us to disentangle different types of SIDS. We demonstrate the potential for HIDS to cause unexpected or undesirable behavior in these environments, and propose and test a mitigation strategy.
The main idea of the paper: When using meta-learning there is an inherent incentive for the learner to win by making the task easier. The authors generalise this effect to a larger class of problems where the learning framework induces a set of Hidden Incentive for Distributional Shift (HIDS) and introduce Context Swapping, a HIDS mitigation technique. In the experimental section, the authors propos a HIDS unit test which then they employ to show that PBT (Population Based-Trainng), a popular meta-learning algorithm exhibits HIDS ant that context swapping helps fixing it. Overall, I found the idea of the paper interesting, but the attempt to generalise the effect from meta-learning to general learning setups hard to follow and detracting from the overall value. I think the authors should have restricted their claims to PBT and PBT-like methods and follow-up with something more general in future work. Furthermore, the notation and formaliation of the problem are incomplete: * the concept of ‘trajectory’ is introduced without being properly defined, though its crucial in the definition of the proposed HIDS mitigation approach * the context swapping algorithm description is not clearly motivated and explained, a diagram showing the learner shuffling would be quite helpful * in the HIDS unit-test section, the game theoretical setup is only partially explained, the defection and cooperation actions are not clearly linked to the HIDS * In Section 4.1.1 HIDS UNIT TEST EXPERIMENTAL RESULTS AND DISCUSSION Figure2 is refered for results without ever stating the task and the Figure itself does not mention it In terms of suggestions, I think the paper needs to go through a careful refactoring with an attention to technical details (careful concept definition, introduction of notation, clarity on experimental setup)?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Hidden incentives for self-induced distributional shift
Decisions made by machine learning systems have increasing influence on the world. Yet it is common for machine learning algorithms to assume that no such influence exists. An example is the use of the i.i.d. assumption in online learning for applications such as content recommendation, where the (choice of) content displayed can change users* perceptions and preferences, or even drive them away, causing a shift in the distribution of users. Generally speaking, it is possible for an algorithm to change the distribution of its own inputs. We introduce the term self-induced distributional shift (SIDS) to describe this phenomenon. A large body of work in reinforcement learning and causal machine learning aims to deal with distributional shift caused by deploying learning systems previously trained offline. Our goal is similar, but distinct: we point out that changes to the learning algorithm, such as the introduction of meta-learning, can reveal hidden incentives for distributional shift (HIDS), and aim to diagnose and prevent problems associated with hidden incentives. We design a simple environment as a *unit test* for HIDS, as well as a content recommendation environment which allows us to disentangle different types of SIDS. We demonstrate the potential for HIDS to cause unexpected or undesirable behavior in these environments, and propose and test a mitigation strategy.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The main idea of the paper: When using meta-learning there is an inherent incentive for the learner to win by making the task easier. The authors generalise this effect to a larger class of problems where the learning framework induces a set of Hidden Incentive for Distributional Shift (HIDS) and introduce Context Swapping, a HIDS mitigation technique. In the experimental section, the authors propos a HIDS unit test which then they employ to show that PBT (Population Based-Trainng), a popular meta-learning algorithm exhibits HIDS ant that context swapping helps fixing it. Overall, I found the idea of the paper interesting, but the attempt to generalise the effect from meta-learning to general learning setups hard to follow and detracting from the overall value. I think the authors should have restricted their claims to PBT and PBT-like methods and follow-up with something more general in future work. Furthermore, the notation and formaliation of the problem are incomplete: * the concept of ‘trajectory’ is introduced without being properly defined, though its crucial in the definition of the proposed HIDS mitigation approach * the context swapping algorithm description is not clearly motivated and explained, a diagram showing the learner shuffling would be quite helpful * in the HIDS unit-test section, the game theoretical setup is only partially explained, the defection and cooperation actions are not clearly linked to the HIDS * In Section 4.1.1 HIDS UNIT TEST EXPERIMENTAL RESULTS AND DISCUSSION Figure2 is refered for results without ever stating the task and the Figure itself does not mention it In terms of suggestions, I think the paper needs to go through a careful refactoring with an attention to technical details (careful concept definition, introduction of notation, clarity on experimental setup)?
assistant
The main idea of the paper: When using meta-learning there is an inherent incentive for the learner to win by making the task easier. The authors generalise this effect to a larger class of problems where the learning framework induces a set of Hidden Incentive for Distributional Shift (HIDS) and introduce Context Swapping, a HIDS mitigation technique. In the experimental section, the authors propos a HIDS unit test which then they employ to show that PBT (Population Based-Trainng), a popular meta-learning algorithm exhibits HIDS ant that context swapping helps fixing it. Overall, I found the idea of the paper interesting, but the attempt to generalise the effect from meta-learning to general learning setups hard to follow and detracting from the overall value. I think the authors should have restricted their claims to PBT and PBT-like methods and follow-up with something more general in future work. Furthermore, the notation and formaliation of the problem are incomplete: * the concept of ‘trajectory’ is introduced without being properly defined, though its crucial in the definition of the proposed HIDS mitigation approach * the context swapping algorithm description is not clearly motivated and explained, a diagram showing the learner shuffling would be quite helpful * in the HIDS unit-test section, the game theoretical setup is only partially explained, the defection and cooperation actions are not clearly linked to the HIDS * In Section 4.1.1 HIDS UNIT TEST EXPERIMENTAL RESULTS AND DISCUSSION Figure2 is refered for results without ever stating the task and the Figure itself does not mention it In terms of suggestions, I think the paper needs to go through a careful refactoring with an attention to technical details (careful concept definition, introduction of notation, clarity on experimental setup)?