In this paper, we aim at tackling a general issue in NLP tasks where some of the negative examples are highly similar to the positive examples, i.e., hard-negative examples). We propose the distant supervision as a regularizer (DSReg) approach to tackle this issue. We convert the original task to a multi-task learning problem, in which we first utilize the idea of distant supervision to retrieve hard-negative examples. The obtained hard-negative examples are then used as a regularizer, and we jointly optimize the original target objective of distinguishing positive examples from negative examples along with the auxiliary task objective of distinguishing soften positive examples (comprised of positive examples and hard-negative examples) from easy-negative examples. In the neural context, this can be done by feeding the final token representations to different output layers. Using this unbelievably simple strategy, we improve the performance of a range of different NLP tasks, including text classification, sequence labeling and reading comprehension.
This paper proposes to improve performance of NLP tasks by focusing on negative examples that are similar to positive examples (e.g. hard negatives). This is achieved by regularizing the model using extra output classifiers trained to classify examples into up to three classes: positive, negative-easy, and negative-hard. Since those labels are not provided in the original data, examples are classified using heuristics (e.g. negative examples that contain a lot of features predictive of a positive class will be considered as hard-negative examples), which are used to provide distant supervision. This general approach is evaluated on phrase classification tasks, one information extraction task, and one MRCQA task.
Although the proposed approach is interesting, this paper has several weaknesses (i) the method is not sufficiently justified or analyzed; (ii) there are missing links with previous work (notably on domain adversarial training); (iii) experimental setting is rather weak.
1) About the justification of the approach:
1.1) I feel like the proposed are not intuitively justified enough. They point out that "L_2 can be thought as an objective to capture the shared features in positive examples and hard-negative examples". Why would that be good from an intuitive perspective ?
1.2) L_3 is forcing the model to group all the hard-negative examples together. Do you have an intuition why would that be useful ?
1.3) What happens if the model overfits the hard negative examples in the training set ? This would mean that it has captured some features that can distinguish positive / negative label. Why would L_3 help in that case ?
2) About related approaches:
2.1) How does this method relate to domain adversarial training applied to positive and hard-negatives and adversarial examples in general ?
2.2) Would similar performance be obtained by virtual adversarial training for example ?
3) About the experimental setting:
3.1) The performance reported is well below the use of recent work on these datasets and recent models such as BERT. Would these improvements carry over to bigger architectures ?
3.2) In SST, the paper say they use BERT large, but the baseline performance (81.5) is well below BERT large performance in the original paper (94.9, https://arxiv.org/pdf/1810.04805.pdf). Why the mismatch ?
3.3) What's the proportion of hard-negative examples mined for training and test set ? While the heuristics used seem reasonable, without those numbers, it is impossible to know if the heuristics truly predict hard-negative examples.
3.4) Does the performance gain comes from better predicting hard-negative examples in the test set ? One could analyze the performance per error type (i.e. true positive, false negative, false positive (easy), false positive (hard)) with the baseline model and of the various proposed regularizing tasks (e.g. L_2 and L_3, both in training and test).
3.5) The heuristics are used to pick what should be adversarial examples, but there is no mentions of this concept in the text. Oversampling those adversarial examples could, potentially, improve the performance of the baseline model. It would be interesting to try this.
3.6) If possible, it would be good to add standard deviation to the results obtained running multiple runs.
3.7) The visualization sub-section is anecdotal and not especially illuminating, and its text seems to refer to a different example than the figures ("interesting" is not in the figures.)
Minor points:
- Section 3 (Models) may be made shorter, the models used are utterly simple. This could free up space for more experiments.
- In the tables, simply adding the name of the used model to the "L1" rows would be clearer.
- The description of the pipelined results in Section 4.1 does not match the results shown in the table.
- The citations are not well integrated with the text (\citep vs \cite), and the formatting of CRF changes in the last paragraph of Section 3.2.