While deep neural networks have achieved impressive performance on a range of NLP tasks, these data-hungry models heavily rely on labeled data. To make the most of each example, previous work has introduced natural language (NL) explanations to serve as supplements to mere labels. Such NL explanations can provide sufficient domain knowledge for generating more labeled data over new instances, while the annotation time only doubles. However, directly applying the NL explanations for augmenting model learning encounters two challenges. First, NL explanations are unstructured and inherently compositional, which asks for a modularized model to represent their semantics. Second, NL explanations often have large numbers of linguistic variants, resulting in low recall and limited generalization ability when applied to unlabeled data. In this paper, we propose a novel Neural Modular Execution Tree (NMET) framework for augmenting sequence classification with NL explanations. First, we transform NL explanations into executable logical forms using a semantic parser. NMET then employs a neural module network architecture to generalize the different types of actions (specified by the logical forms) for labeling data instances and accumulates the results with soft logic, which substantially increases the coverage of each NL explanation. Experiments on two NLP tasks, relation extraction, and sentiment analysis, demonstrate its superiority over baseline methods by leveraging NL explanation. Its extension to multi-hop question answering achieves performance gain with light annotation effort. Also, NMET achieves much better performance compared to traditional label-only supervised models in the same annotation time.
One recent work that comes to mind from ACL 2019: Leveraging Language Models for Commonsense Reasoning (Rajani et al 2019). In that work, they also have human annotators provide explanations (extending the CommonsenseQA dataset), and they show that by training with these explanations, inference is improved even without them. They also train a language model to generate the explanations, and they show that the language model generated explanations improve performance further at inference time. Seems like a reasonable reference to contrast the more structured approach to using explanations like Srivasta et al (2017), Hancock et al (2018), and this work.
I find the method summary beginning with "Human explanations are first converted to machine-actionable logical forms by a semantic parser" until the end of that paragraph to be unnecessary. Actually, as I read it, I find myself asking a lot of questions that get answered below. So I would prefer scrapping that method summary and just getting straight into the Explanation Parsing.
"indicates the the logical form matches" redundant 'the'
I can't find a definition for LF(E) anywhere, and yet LF(E) is present in many tables. I see that E is mentioned to be the explanations, but this is only in the caption of Table 2 even though the symbol is first used in the first paragraph of Section 4.1. I'm assuming LF logical forms applied directly to explanations, but this should be stated explicitly. Can you elaborate on why it is so dominant on precision In Table 6 and 7 of the Appendix, but low on recall, rather than just saying this is expected in Section 4.1? 
"For keyword query q, we directly encode it into vector z_q by bi-LSTM and attention." Can you elaborate on how z_q is constructed? Is it the final state of a forward LSTM concatenated with the final state of a backward LSTM?
Why is it so essential that you study the setting in which explanations are low-resource? I'm curious to see what would happen with more explanations.
I am surprised that none of the modules or compared methods include any architecture that use a Transformer or a form of contextualized word vectors (McCann et al 2017, Peters et al 2017, Devlin et al 2018). Is there an explanation for this?
I would prefer to see a larger suite of tested tasks given that each of these datasets is quite small. Would any other tasks from benchmarks like GLUE or SuperGLUE be amenable to your approach? The tasks you've chosen limit the scope of this work and leaves the question of whether it would generally improve across a greater variety of tasks, especially tasks that have seen significant improvement using new methods. Your claim would be much stronger if the explanations were shown to be helpful even to pretrained models like BERT when fine-tuned for a specific task. In particular, it would be interesting to see how the benefits of explanations vary for different kinds of tasks and for different training set sizes. 
"In the real world, a more realistic problem is that, with limited human-power, should we just annotate more labels or spend time explaining existing annotations."" I think that you mean "should we" makes it sound like this is a question, but there is no question mark, and it makes more sense as a statement. I propose "...human-power, there is a question of whether it is more valuable to ..."
"Explanations prove to be an very efficient form for data annotation." should be "a very" not "an very".
" a vital rule" should be "a vital role"
I find the first paragraph of Section 4.3 quite abstruse and bare in its explanation of the method used.