Published as a conference paper at ICLR 2018 BEYOND WORD IMPORTANCE: CONTEXTUAL DE- COMPOSITION TO EXTRACT INTERACTIONS FROM LSTMS
The driving force behind the recent success of LSTMs has been their ability to learn complex and non-linear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm for analysing individual predictions made by standard LSTMs, without any changes to the underlying model. By decomposing the output of a LSTM, CD captures the contributions of combinations of words or variables to the final prediction of an LSTM. On the task of sentiment analysis with the Yelp and SST data sets, we show that CD is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTM's final prediction. Using the phrase-level labels in SST, we also demonstrate that CD is able to successfully extract positive and negative negations from an LSTM, something which has not previously been done.
In this paper, the authors propose a new LSTM variant that allows to produce interpretations by capturing the contributions of the words to the final prediction and the way their learned representations are combined in order to yield that prediction. They propose a new approach that they call Contextual Decomposition (CD). Their approach consists of disambiguating interaction between LSTM’s gates where gates are linearized so the products between them is over linear sums of contributions from different factors. The hidden and cell states are also decomposed in terms of contributions to the “phrase” in question and contributions from elements outside of the phrase. The motivation of the proposed decomposition using LSTMs is that the latter are powerful at capturing complex non-linear interactions, so, it would be useful to observe how these interactions are handled and to interpret the LSTM’s predictions. As the authors intention is to build a way of interpreting LSTMs output and not to increase the model’s accuracy, the empirical results illustrate the ability of their decomposition of giving a plausible interpretation to the elements of a sentence. They compare their method with different existing method by illustrating samples from the Yelp Polarity and SST datasets. They also show the ability of separating the distribution of CD scores related to positive and negative phrases on respectively Yelp Polarity and SST.
The proposed approach is potentially of great benefit as it is simple and elegant and could lead to new methods in the same direction of research. The sample illustrations, the scatter plots and the CD score distributions are helpful to asses the benefit of the proposed approach.
The writing could be improved as it contains parts where it leads to confusion. The details related to the linearization (section 3.2.2), the training (4.1) could be improved. In equation 25, it is not clear what  π_{i}^(-1) and x_{π_{i}} represent but the example in equation 26 makes it clearer. The section would be clearer if each index and notation used is explained explicitly.
(CD in known for Contrastive Divergence in the deep learning community. It would be better if Contextual Decomposition is not referred by CD.)
Training details are given in section 4.1 where the authors mention the integrated gradient baseline without mentioning the reference paper to it (however they do mention the reference paper at each of table 1 and 2). it would be clearer for the reader if the references are also mentioned in section 4.1 where integrated gradient is introduced. Along with the reference, a brief description of that baseline could be given. 
The “Leave one out” baseline is never mentioned in text before section 4.4 (and tables 1 and 2). Neither the reference nor the description of the baseline are given. It would have been clearer to the reader if this had been the case.
Overall, the paper contribution is of great benefit. The quality of the paper could be improved if the above explanations and details are given explicitly in the text.