Supervised models suffer from domain shifting where distribution mismatch across domains greatly affect model performance. Particularly, noise scattered in each domain has played a crucial role in representing such distribution, especially in various natural language processing (NLP) tasks. In addressing this issue, training data selection (TDS) has been proven to be a prospective way to train supervised models with higher performance and efficiency. Following the TDS methodology, in this paper, we propose a general data selection framework with representation learning and distribution matching simultaneously for domain adaptation on neural models. In doing so, we formulate TDS as a novel selection process based on a learned selection distribution from the input data, which is produced by a trainable selection distribution generator (SDG) that is optimized by reinforcement learning (RL). Then, the model trained by the selected data not only predicts the target domain data in a specific task, but also provides input for the value function of the RL, which exactly helps update the parameters of SDG in a policy gradient way. Experiments are conducted on three typical NLP tasks, namely, part-of-speech tagging, dependency parsing, and sentiment analysis. Results demonstrate the validity and effectiveness of our approach.
Response to author comments:
Unfortunately I am still significantly unclear on why RL is useful here.  The author response attempts to clarify that by pointing me to paragraph 2 of the intro, which states that RL has been used for data selection in other settings in the past.  What would help me (and I believe, the paper) more is a reason why greedy selection isn't sufficient for this particular problem.  Even just a single motivating example would be extremely helpful.  R3 mentioned similar concerns in their review, saying that the paper lacks explanation for why RL would win over non-RL for e.g. sentiment analysis.
Likewise, while I appreciate the authors comparing against a stronger baseline in Figure 3, I don't know how to interpret the figure.  Why is Figure 3(b) better than Figure 3(c), and why does using RL cause that difference to arise?
Original review:
Domain adaptation is an interesting task, and new methods for it would be welcome.  This paper appears to have technical depth and the experimental results are promising.  However, the presented approach is complex, and I found it very hard to understand -- both in terms of how exactly it works, and in terms of why the chosen techniques were chosen.  More detail on my questions and confusions follows.
First, I never understood the motivation for using RL here.  If minimizing the distance between selected data from the source domain and data in the target domain is the objective (equation 1), how does RL help?  The reward seems like it is immediate in each time step.  How does the *order* in which we add source examples to our collection matter?  I never understood the crucial difference that made the RL approach outperform the baselines that just select examples that minimize e.g. JS divergence.  Neither the paper's discussion of motivation nor the experimental analysis clarifies this.
The paper says in Section 2.1 that a formal description of the representations is to follow.  I didn't see this description (I do not see a formal definition of how the feature extractor works, and e.g. how it produces vectors that are *distributions* that can be used within e.g. JS divergence).
The paper also says it follows (Ruder and Plank, 2017) in using JS as a baseline, but as I understand that work the JS baseline is computed over words, not learned representations.  What is done in the submission, is the JS baseline over words in the instances, or the representations from the feature extractor?
What is the reason for partitioning the source data into disjoint "data bags"?  Why not just select the best source domain examples (from among all the source data) using RL?
The experiments are generally over enough tasks and compare against several baselines, and although the empirical wins are not that large I feel that they would be sufficient for publication if not for my other concerns.  The analysis (sec 4) did not make it clear to me why the RL approach works.  The visualization in Figure 3 only contrasts the proposed approach with a weak baseline of selecting all source data -- what we really need to see is an analysis that reveals why the learning of a policy with RL is better than simply greedily minimizing JS for each source data selection, for up to some limit of n selections.
Minor
The paper has a number of typos
The citations in the paper are mis-formatted -- seem to use shortcite where they shouldn't (e.g. "scenarios Akopyan and Khashba (2017)" should be "scenarios (Akopyan and Khashba, 2017)").
When the policy \pi_w(a | s) is introduced at the start of Sec 2.2, it uses symbols (a, s) that have not been defined, also that policy variable is not really utilized in the text so it could be deleted.