The ability to predict matches between two sources of text has a number of applications including natural language inference (NLI) and question answering (QA). While flexible neural models have become effective tools in solving these tasks, they are rarely transparent in terms of the mechanism that mediates the prediction. In this paper, we propose a self-explaining architecture where the model is forced to highlight, in a dependent manner, how spans of one side of the input match corresponding segments of the other side in order to arrive at the overall decision. The text spans are regularized to be coherent and concise, and their correspondence is captured explicitly. The text spans -rationales -are learned entirely as latent mechanisms, guided only by the distal supervision from the end-to-end task. We evaluate our model on both NLI and QA using three publicly available datasets. Experimental results demonstrate quantitatively and qualitatively that our method delivers interpretable justification of the prediction without sacrificing state-of-the-art performance. Our code and data split will be publicly available.
This paper tackles the problem of generating rationales for text matching problems (i.e., two pieces of text are given). The approach is in a similar spirit as (Lei et al, 2016) while the latter mainly focuses on one piece of text for text classification problems and this work focuses on generating pairs of rationales. The approach has been evaluated on NLI and QA datasets, which demonstrates that the generated rationales are sensible and comes at a cost of accuracy.
The approach employs a generation-encoding-generation schema: it firsts generates the rationale from one side as a sequence tagging problem, re-encodes the rationales and predicts the rationale on the other side as a span prediction problem. Leveraging a match-LSTM framework and generated rationales for prediction, the model can be trained using a policy gradient method.
Overall, I think this problem is novel and interesting. However, I am not fully convinced whether the proposed solution (and its implementation) is the right way to do so. Also, the paper writing needs to much improved.
First of all, there is certainly a drop in the end task performance while it is unclear whether the derived rationales are really that useful (if the goal is interpretability) in the current evaluation. I am not convinced by the noisy SciTrail evaluation for rationales -- the noisy part p’ can be totally irrelevant and assume that the rationale generation component learns some sort of alignment between two parts, so it is not surprising that the model will not select words from p’ and it doesn’t really show that the rationales are useful. I think it is necessary to conduct some human evaluation for generate rationales and also provide some simple baselines for comparison (for example, just converting the soft-attention in math-LSTM to some hard selections) and see if this interpretability (at a cost of task performance) is really worthy or not.
Secondly, I am not sure that whether the current way of generating the rationale pairs really makes sense or not.
It casts the rationale generation on one side as a tagging problem while the rationale generation on the other side as a span prediction problem. Why is that? Do you make any assumption that the two pieces of texts are not symmetric (e.g., one side is much longer than the other side like most of the current QA setup)?
There is a regularization term for both x and y but it seems that there isn’t any constraint that the generated rationales on the y side are not overlapping. Is it a problem or not? I don’t know how this is dealt with in the implementation.
Understanding sec 3 takes some efforts and I think the presentation could be much improved. For example, q * {x^k} is not defined -- I assume it means extracting the subset of q based on the 1’s in {x^k}. The equations in Sec 3.2 can be made clearer.
Finally, it is also unclear that how the 3 datasets were chosen. There are so many NLI and QA datasets (some of them are more popular and more competitive) at this point. Is there a reason that these datasets were chosen? There is a setup called ‘no rationalization w/ re-encoding’ which means that the rationale is already provided on one side, but is unclear that whether the OpenIE tuple and the searchQA queries can be used as rationales directly.
Minor points:
- Distal supervision -> distant supervision
- The first paragraph of Introduction, “absent attention or rationale mechanisms”, what does it mean by ‘absent attention’? Isn’t it the case that all the models used attention mechanisms?