Quantifying the value of data is a fundamental problem in machine learning. Data valuation has multiple important use cases: (1) building insights about the learning task, (2) domain adaptation, (3) corrupted sample discovery, and (4) robust learning. To adaptively learn data values jointly with the target task predictor model, we propose a meta learning framework which we name Data Valuation using Reinforcement Learning (DVRL). We employ a data value estimator (modeled by a deep neural network) to learn how likely each datum is used in training of the predictor model. We train the data value estimator using a reinforcement signal of the reward obtained on a small validation set that reflects performance on the target task. We demonstrate that DVRL yields superior data value estimates compared to alternative methods across different types of datasets and in a diverse set of application scenarios. The corrupted sample discovery performance of DVRL is close to optimal in many regimes (i.e. as if the noisy samples were known apriori), and for domain adaptation and robust learning DVRL significantly outperforms state-of-the-art by 14.6% and 10.8%, respectively.
This paper proposes a meta learning approach based on data valuation for reinforcement learning tasks. The core idea is to train a second network (the data value estimator) in conjunction to a regular predictor network. The predictor is then trained with samples chosen via the data value estimation. The authors motivate this construction with the goal to filter out unreliable and corrupted data. 
It's well established that RL poses a difficult learning problem, and as such the goal to improve the RL process is definitely a good one. To the best of my knowledge the approach proposed here is new. The exposition of the paper is also quite clear, and all parts of the approach are explained nicely. In addition, the submission contains a thorough evaluation of the method. 
A central point for the method seems to be the validation data set which is used to train the data value estimator. The text emphasizes that this data set can be "small" several times, and the discussion and results of section 4.5 try to shed light here. However, Figure 5 indicates that a fairly large fraction of samples is needed to identify, e.g., more than 50% of the corrupted samples.
Another cricital aspect for meta-learning approaches such as this one is also the training time. RL is already expensive, so if the meta learning introduces a large factor, the training could quickly become infeasible. Here, the text gives a factor of about 3, which is noticeable, but not overly big. This still seems practical. Potentially, the estimator could also be reused (at least partially) for repeated training runs.
The tests in figure 2 are somewhat unituitive at first - I was expecting results of models trained on datasets with different samples being removed beforehand, rather than removing samples based on a trained estimator. However, this test makes sense on second sight, and e.g., the significant drop in performance after removing the most important samples indicates that the estimator was able to correctly identify a certain portion of data that is actually important for successful predictions. In addition to the noisy label and domain adaptation tests, this paints a positive picture. The method seems to yield useful (be it somewhat small) improvements in terms of learning performance.
One aspect that could be justified better in my opinion is the choise for a discrete representation for the data value estimator. Shouldn't the method likewise work with a continuous representation here? The paper explain how the discrete model is trained with quite some detail, but the choice itself could be motivated more clearly.
However, overall I think the method is interesting and yields nice performance improvements. It is described and evaluated in detail, so I think the paper could me included in the ICLR program.