Automatic Rule Extraction from Long Short Term Memory Networks
Although deep learning models have proven effective at solving problems in natural language processing, the mechanism by which they come to their conclusions is often unclear. As a result, these models are generally treated as black boxes, yielding no insight of the underlying learned patterns. In this paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new approach for tracking the importance of a given input to the LSTM for a given output. By identifying consistently important patterns of words, we are able to distill state of the art LSTMs on sentiment analysis and question answering into a set of representative phrases. This representation is then quantitatively validated by using the extracted phrases to construct a simple, rule-based classifier which approximates the output of the LSTM.
This work proposes a pattern extraction method to both understand what a trained LSTM has learnt and to allow implementation of a hand-coded algorithm that performs similarly to the LSTM. Good results are shown on one dataset for one model architecture so it is unclear how well this approach will generalize, however, it seems it will be a useful way to understand and debug models. The questions in WikiMovies seem to be generated from templates and so this pattern matching approach will likely work well. However, from the experiments it*s not clear if this will extend to other types of Q&A tasks where the answer may be free form text and not be a substring in the document. Is the model required to produce a continuous span over the original document? The approach also seems to have some deficiencies in how it handles word types such as numbers or entity names. This can be encoded in the embedding for the word but from the description of the algorithm, it seems that the approach requires an entity detector. Does this mean that the approach is unable to determine when it has reached an entity from the decomposition of the output of the LSTM? The results where *manual pattern matching* where explicit year annotations are used, seem to show that the automatic method is unable to deal with word types. It would also be good to see an attention model as a baseline in addition to the gradient-based baseline. Minor comments: - P and Q seem to be undefined. - Some references seem to be bad, e.g. in section 5.1: *in 1* instead of *in table 1*. Similarly above section 7: *as shown in 3* and in section 7.1. - In the paragraph above section 6.3: *adam* -> *Adam*.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Automatic Rule Extraction from Long Short Term Memory Networks
Although deep learning models have proven effective at solving problems in natural language processing, the mechanism by which they come to their conclusions is often unclear. As a result, these models are generally treated as black boxes, yielding no insight of the underlying learned patterns. In this paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new approach for tracking the importance of a given input to the LSTM for a given output. By identifying consistently important patterns of words, we are able to distill state of the art LSTMs on sentiment analysis and question answering into a set of representative phrases. This representation is then quantitatively validated by using the extracted phrases to construct a simple, rule-based classifier which approximates the output of the LSTM.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work proposes a pattern extraction method to both understand what a trained LSTM has learnt and to allow implementation of a hand-coded algorithm that performs similarly to the LSTM. Good results are shown on one dataset for one model architecture so it is unclear how well this approach will generalize, however, it seems it will be a useful way to understand and debug models. The questions in WikiMovies seem to be generated from templates and so this pattern matching approach will likely work well. However, from the experiments it*s not clear if this will extend to other types of Q&A tasks where the answer may be free form text and not be a substring in the document. Is the model required to produce a continuous span over the original document? The approach also seems to have some deficiencies in how it handles word types such as numbers or entity names. This can be encoded in the embedding for the word but from the description of the algorithm, it seems that the approach requires an entity detector. Does this mean that the approach is unable to determine when it has reached an entity from the decomposition of the output of the LSTM? The results where *manual pattern matching* where explicit year annotations are used, seem to show that the automatic method is unable to deal with word types. It would also be good to see an attention model as a baseline in addition to the gradient-based baseline. Minor comments: - P and Q seem to be undefined. - Some references seem to be bad, e.g. in section 5.1: *in 1* instead of *in table 1*. Similarly above section 7: *as shown in 3* and in section 7.1. - In the paragraph above section 6.3: *adam* -> *Adam*.
assistant
This work proposes a pattern extraction method to both understand what a trained LSTM has learnt and to allow implementation of a hand-coded algorithm that performs similarly to the LSTM. Good results are shown on one dataset for one model architecture so it is unclear how well this approach will generalize, however, it seems it will be a useful way to understand and debug models. The questions in WikiMovies seem to be generated from templates and so this pattern matching approach will likely work well. However, from the experiments it*s not clear if this will extend to other types of Q&A tasks where the answer may be free form text and not be a substring in the document. Is the model required to produce a continuous span over the original document? The approach also seems to have some deficiencies in how it handles word types such as numbers or entity names. This can be encoded in the embedding for the word but from the description of the algorithm, it seems that the approach requires an entity detector. Does this mean that the approach is unable to determine when it has reached an entity from the decomposition of the output of the LSTM? The results where *manual pattern matching* where explicit year annotations are used, seem to show that the automatic method is unable to deal with word types. It would also be good to see an attention model as a baseline in addition to the gradient-based baseline. Minor comments: - P and Q seem to be undefined. - Some references seem to be bad, e.g. in section 5.1: *in 1* instead of *in table 1*. Similarly above section 7: *as shown in 3* and in section 7.1. - In the paragraph above section 6.3: *adam* -> *Adam*.