Optimizing Data Usage via Differentiable Rewards
To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, a machine learning model could potentially be trained better with a scorer that “adapts” to its current learning state and estimates the importance of each training data instance. Training such an adaptive scorer efficiently is a challenging problem; in order to precisely quantify the effect of a data instance at a given time during the training, it is typically necessary to first complete the entire training process. To efficiently optimize data usage, we propose a reinforcement learning approach called Differentiable Data Selection (DDS). In DDS, we formulate a scorer network as a learnable function of the training data, which can be efficiently updated along with the main model being trained. Specifically, DDS updates the scorer with an intuitive reward signal: it should up-weigh the data that has a similar gradient with a dev set upon which we would finally like to perform well. Without significant computing overhead, DDS delivers strong and consistent improvements over several strong baselines on two very different tasks of machine translation and image classification.
This paper presents a reinforcement learning approach towards using data that present best correlation with a validation set’s gradient signal. The broader point of this paper is that there is inevitably some distribution shift going from train to test set - and the validation set can be a small curated set whose distribution is closer to the testing distribution than what the training dataset*s distribution is. The problem setup bears relationship to several areas including domain adaptation/covariate shift problems, curriculum learning based approaches amongst others. One assumption that I see which needs to be understood more is equation (6) - wherein, somehow, there is a Markov assumption used to zero out the contribution of the scoring network on parameters unto previous time step. Trying to understand the implications of this assumption (how the performance varies with/without this assumption) would be instructive for understanding potential shortcomings of this framework. I think the paper is well written, handles an important question. That said, I am not too aware of recent work in this area to make a decisive judgement on this paper’s novelty/contributions.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Optimizing Data Usage via Differentiable Rewards
To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, a machine learning model could potentially be trained better with a scorer that “adapts” to its current learning state and estimates the importance of each training data instance. Training such an adaptive scorer efficiently is a challenging problem; in order to precisely quantify the effect of a data instance at a given time during the training, it is typically necessary to first complete the entire training process. To efficiently optimize data usage, we propose a reinforcement learning approach called Differentiable Data Selection (DDS). In DDS, we formulate a scorer network as a learnable function of the training data, which can be efficiently updated along with the main model being trained. Specifically, DDS updates the scorer with an intuitive reward signal: it should up-weigh the data that has a similar gradient with a dev set upon which we would finally like to perform well. Without significant computing overhead, DDS delivers strong and consistent improvements over several strong baselines on two very different tasks of machine translation and image classification.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents a reinforcement learning approach towards using data that present best correlation with a validation set’s gradient signal. The broader point of this paper is that there is inevitably some distribution shift going from train to test set - and the validation set can be a small curated set whose distribution is closer to the testing distribution than what the training dataset*s distribution is. The problem setup bears relationship to several areas including domain adaptation/covariate shift problems, curriculum learning based approaches amongst others. One assumption that I see which needs to be understood more is equation (6) - wherein, somehow, there is a Markov assumption used to zero out the contribution of the scoring network on parameters unto previous time step. Trying to understand the implications of this assumption (how the performance varies with/without this assumption) would be instructive for understanding potential shortcomings of this framework. I think the paper is well written, handles an important question. That said, I am not too aware of recent work in this area to make a decisive judgement on this paper’s novelty/contributions.
assistant
This paper presents a reinforcement learning approach towards using data that present best correlation with a validation set’s gradient signal. The broader point of this paper is that there is inevitably some distribution shift going from train to test set - and the validation set can be a small curated set whose distribution is closer to the testing distribution than what the training dataset*s distribution is. The problem setup bears relationship to several areas including domain adaptation/covariate shift problems, curriculum learning based approaches amongst others. One assumption that I see which needs to be understood more is equation (6) - wherein, somehow, there is a Markov assumption used to zero out the contribution of the scoring network on parameters unto previous time step. Trying to understand the implications of this assumption (how the performance varies with/without this assumption) would be instructive for understanding potential shortcomings of this framework. I think the paper is well written, handles an important question. That said, I am not too aware of recent work in this area to make a decisive judgement on this paper’s novelty/contributions.