Several deep learning models have been proposed for question answering. However, due to their single-pass nature, they have no way to recover from local maxima corresponding to incorrect answers. To address this problem, we introduce the Dynamic Coattention Network (DCN) for question answering. The DCN first fuses co-dependent representations of the question and the document in order to focus on relevant parts of both. Then a dynamic pointing decoder iterates over potential answer spans. This iterative procedure enables the model to recover from initial local maxima corresponding to incorrect answers. On the Stanford question answering dataset, a single DCN model improves the previous state of the art from 71.0% F1 to 75.9%, while a DCN ensemble obtains 80.4% F1.
Several deep learning models have been proposed for question answering. However, due to their single-pass nature, they have no way to recover from local maxima corresponding to incorrect answers. To address this problem, we introduce the Dynamic Coattention Network (DCN) for question answering. The DCN first fuses co-dependent representations of the question and the document in order to focus on relevant parts of both. Then a dynamic pointing decoder iterates over potential answer spans. This iterative procedure enables the model to recover from initial local maxima corresponding to incorrect answers. On the Stanford question answering dataset, a single DCN model improves the previous state of the art from 71.0% F1 to 75.9%, while a DCN ensemble obtains 80.4% F1.
1 INTRODUCTION
Question answering (QA) is a crucial task in natural language processing that requires both natural language understanding and world knowledge. Previous QA datasets tend to be high in quality due to human annotation, but small in size (Berant et al., 2014; Richardson et al., 2013). Hence, they did not allow for training data-intensive, expressive models such as deep neural networks.
To address this problem, researchers have developed large-scale datasets through semi-automated techniques (Hermann et al., 2015; Hill et al., 2016). Compared to their smaller, hand-annotated counterparts, these QA datasets allow the training of more expressive models. However, it has been shown that they differ from more natural, human annotated datasets in the types of reasoning required to answer the questions (Chen et al., 2016).
Recently, Rajpurkar et al. (2016) released the Stanford Question Answering dataset (SQuAD), which is orders of magnitude larger than all previous hand-annotated datasets and has a variety of qualities that culminate in a natural QA task. SQuAD has the desirable quality that answers are spans in a reference document. This constrains answers to the space of all possible spans. However, Rajpurkar et al. (2016) show that the dataset retains a diverse set of answers and requires different forms of logical reasoning, including multi-sentence reasoning.
We introduce the Dynamic Coattention Network (DCN), illustrated in Fig. 1, an end-to-end neural network for question answering. The model consists of a coattentive encoder that captures the interactions between the question and the document, as well as a dynamic pointing decoder that alternates between estimating the start and end of the answer span. Our single model obtains an F1 of 75.9% compared to the best published result of 71.0% (Yu et al., 2016). In addition, our ensemble model obtains an F1 of 80.4% compared to the second best result of 78.1% on the official SQuAD leaderboard.1
∗Equal contribution 1As of Nov. 3 2016. See https://rajpurkar.github.io/SQuAD-explorer/ for latest results.
2 DYNAMIC COATTENTION NETWORKS
Figure 1 illustrates an overview of the DCN. We first describe the encoders for the document and the question, followed by the coattention mechanism and the dynamic decoder which produces the answer span.
2.1 DOCUMENT AND QUESTION ENCODER
Let (xQ1 , x Q 2 , . . . , x Q n ) denote the sequence of word vectors corresponding to words in the question and (xD1 , x D 2 , . . . , x D m) denote the same for words in the document. Using an LSTM (Hochreiter
& Schmidhuber, 1997), we encode the document as: dt = LSTMenc ( dt−1, x D t ) . We define the document encoding matrix as D = [d1 . . . dm d∅] ∈ R`×(m+1). We also add a sentinel vector d∅ (Merity et al., 2016), which we later show allows the model to not attend to any particular word in the input.
The question embeddings are computed with the same LSTM to share representation power: qt = LSTMenc ( qt−1, x Q t ) . We define an intermediate question representation Q′ = [q1 . . . qn q∅] ∈
R`×(n+1). To allow for variation between the question encoding space and the document encoding space, we introduce a non-linear projection layer on top of the question encoding. The final representation for the question becomes: Q = tanh ( W (Q)Q′ + b(Q) ) ∈ R`×(n+1).
2.2 COATTENTION ENCODER
We propose a coattention mechanism that attends to the question and document simultaneously, similar to (Lu et al., 2016), and finally fuses both attention contexts. Figure 2 provides an illustration of the coattention encoder.
We first compute the affinity matrix, which contains affinity scores corresponding to all pairs of document words and question words: L = D>Q ∈ R(m+1)×(n+1). The affinity matrix is normalized row-wise to produce the attention weights AQ across the document for each word in the question, and column-wise to produce the attention weights AD across the question for each word in the document:
AQ = softmax (L) ∈ R(m+1)×(n+1) and AD = softmax ( L> ) ∈ R(n+1)×(m+1) (1)
Next, we compute the summaries, or attention contexts, of the document in light of each word of the question.
CQ = DAQ ∈ R`×(n+1). (2)
We similarly compute the summaries QAD of the question in light of each word of the document. Similar to Cui et al. (2016), we also compute the summaries CQAD of the previous attention contexts in light of each word of the document. These two operations can be done in parallel, as is shown in Eq. 3. One possible interpretation for the operation CQAD is the mapping of question encoding into space of document encodings.
CD = [ Q;CQ ] AD ∈ R2`×(m+1). (3)
We define CD, a co-dependent representation of the question and document, as the coattention context. We use the notation [a; b] for concatenating the vectors a and b horizontally.
The last step is the fusion of temporal information to the coattention context via a bidirectional LSTM: ut = Bi-LSTM ( ut−1, ut+1, [ dt; c D t ]) ∈ R2`. (4)
We define U = [u1, . . . , um] ∈ R2`×m , which provides a foundation for selecting which span may be the best possible answer, as the coattention encoding.
2.3 DYNAMIC POINTING DECODER
Due to the nature of SQuAD, an intuitive method for producing the answer span is by predicting the start and end points of the span (Wang & Jiang, 2016b). However, given a question-document pair, there may exist several intuitive answer spans within the document, each corresponding to a local maxima. We propose an iterative technique to select an answer span by alternating between predicting the start point and predicting the end point. This iterative procedure allows the model to recover from initial local maxima corresponding to incorrect answer spans.
Figure 3 provides an illustration of the Dynamic Decoder, which is similar to a state machine whose state is maintained by an LSTM-based sequential model. During each iteration, the decoder updates its state taking into account the coattention encoding corresponding to current estimates of the start and end positions, and produces, via a multilayer neural network, new estimates of the start and end positions.
Let hi, si, and ei denote the hidden state of the LSTM, the estimate of the position, and the estimate of the end position during iteration i. The LSTM state update is then described by Eq. 5.
hi = LSTM dec ( hi−1, [ usi−1 ;uei−1 ]) (5)
where usi−1 and uei−1 are the representations corresponding to the previous estimate of the start and end positions in the coattention encoding U .
Given the current hidden state hi, previous start position usi−1 , and previous end position uei−1 , we estimate the current start position and end position via Eq. 6 and Eq. 7.
si = argmax t (α1, . . . , αm) (6)
ei = argmax t (β1, . . . , βm) (7)
where αt and βt represent the start score and end score corresponding to the tth word in the document. We compute αt and βt with separate neural networks. These networks have the same architecture but do not share parameters.
Based on the strong empirical performance of Maxout Networks (Goodfellow et al., 2013) and Highway Networks (Srivastava et al., 2015), especially with regards to deep architectures, we propose a Highway Maxout Network (HMN) to compute αt as described by Eq. 8. The intuition behind using such model is that the QA task consists of multiple question types and document topics. These variations may require different models to estimate the answer span. Maxout provides a simple and effective way to pool across multiple model variations.
αt = HMN start ( ut, hi, usi−1 , uei−1 ) (8)
Here, ut is the coattention encoding corresponding to the tth word in the document. HMN start is illustrated in Figure 4. The end score, βt, is computed similarly to the start score αt, but using a separate HMN end.
We now describe the HMN model:
HMN ( ut, hi, usi−1 , uei−1 ) = max ( W (3) [ m (1) t ;m (2) t ] + b(3) ) (9)
r = tanh ( W (D) [ hi;usi−1 ;uei−1 ]) (10)
m (1) t = max ( W (1) [ut; r] + b (1) )
(11)
m (2) t = max
( W (2)m
(1) t + b
(2) )
(12)
where r ∈ R` is a non-linear projection of the current state with parameters W (D) ∈ R`×5`, m(1)t is the output of the first maxout layer with parameters W (1) ∈ Rp×`×3` and b(1) ∈ Rp×`, and m(2)t is the output of the second maxout layer with parameters W (2) ∈ Rp×`×` and b(2) ∈ Rp×`. m(1)t and m(2)t are fed into the final maxout layer, which has parameters W (3) ∈ Rp×1×2`, and b(3) ∈ Rp. p is the pooling size of each maxout layer. The max operation computes the maximum value over the first dimension of a tensor. We note that there is highway connection between the output of the first maxout layer and the last maxout layer.
To train the network, we minimize the cumulative softmax cross entropy of the start and end points across all iterations. The iterative procedure halts when both the estimate of the start position and the estimate of the end position no longer change, or when a maximum number of iterations is reached. Details can be found in Section 4.1
3 RELATED WORK
Statistical QA Traditional approaches to question answering typically involve rule-based algorithms or linear classifiers over hand-engineered feature sets. Richardson et al. (2013) proposed two baselines, one that uses simple lexical features such as a sliding window to match bags of words, and another that uses word-distances between words in the question and in the document. Berant et al. (2014) proposed an alternative approach in which one first learns a structured representation of the entities and relations in the document in the form of a knowledge base, then converts the question to a structured query with which to match the content of the knowledge base. Wang et al. (2015) described a statistical model using frame semantic features as well as syntactic features such as part of speech tags and dependency parses. Chen et al. (2016) proposed a competitive statistical baseline using a variety of carefully crafted lexical, syntactic, and word order features.
Neural QA Neural attention models have been widely applied for machine comprehension or question-answering in NLP. Hermann et al. (2015) proposed an AttentiveReader model with the release of the CNN/Daily Mail cloze-style question answering dataset. Hill et al. (2016) released another dataset steming from the children’s book and proposed a window-based memory network. Kadlec et al. (2016) presented a pointer-style attention mechanism but performs only one attention step. Sordoni et al. (2016) introduced an iterative neural attention model and applied it to cloze-style machine comprehension tasks.
Recently, Rajpurkar et al. (2016) released the SQuAD dataset. Different from cloze-style queries, answers include non-entities and longer phrases, and questions are more realistic. For SQuAD, Wang & Jiang (2016b) proposed an end-to-end neural network model that consists of a Match-LSTM encoder, originally introduced in Wang & Jiang (2016a), and a pointer network decoder (Vinyals et al., 2015); Yu et al. (2016) introduced a dynamic chunk reader, a neural reading comprehension model that extracts a set of answer candidates of variable lengths from the document and ranks them to answer the question.
Lu et al. (2016) proposed a hierarchical co-attention model for visual question answering, which achieved state of the art result on the COCO-VQA dataset (Antol et al., 2015). In (Lu et al., 2016), the co-attention mechanism computes a conditional representation of the image given the question, as well as a conditional representation of the question given the image.
Inspired by the above works, we propose a dynamic coattention model (DCN) that consists of a novel coattentive encoder and dynamic decoder. In our model, instead of estimating the start and end positions of the answer span in a single pass (Wang & Jiang, 2016b), we iteratively update the
start and end positions in a similar fashion to the Iterative Conditional Modes algorithm (Besag, 1986).
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We train and evaluate our model on the SQuAD dataset. To preprocess the corpus, we use the tokenizer from Stanford CoreNLP (Manning et al., 2014). We use as GloVe word vectors pretrained on the 840B Common Crawl corpus (Pennington et al., 2014). We limit the vocabulary to words that are present in the Common Crawl corpus and set embeddings for out-of-vocabulary words to zero. Empirically, we found that training the embeddings consistently led to overfitting and subpar performance, and hence only report results with fixed word embeddings.
We use a max sequence length of 600 during training and a hidden state size of 200 for all recurrent units, maxout layers, and linear layers. All LSTMs have randomly initialized parameters and an initial state of zero. Sentinel vectors are randomly initialized and optimized during training. For the dynamic decoder, we set the maximum number of iterations to 4 and use a maxout pool size of 16. We use dropout to regularize our network during training (Srivastava et al., 2014), and optimize the model using ADAM (Kingma & Ba, 2014). All models are implemented and trained with Chainer (Tokui et al., 2015).
4.2 RESULTS
Evaluation on the SQuAD dataset consists of two metrics. The exact match score (EM) calculates the exact string match between the predicted answer and a ground truth answer. The F1 score calculates the overlap between words in the predicted answer and a ground truth answer. Because a document-question pair may have several ground truth answers, the EM and F1 for a documentquestion pair is taken to be the maximum value across all ground truth answers. The overall metric is then computed by averaging over all document-question pairs. The offical SQuAD evaluation is hosted on CodaLab 2. The training and development sets are publicly available while the test set is withheld.
2https://worksheets.codalab.org
The performance of the Dynamic Coattention Network on the SQuAD dataset, compared to other submitted models on the leaderboard 3, is shown in Table 1. At the time of writing, our singlemodel DCN ranks first at 66.2% exact match and 75.9% F1 on the test data among single-model submissions. Our ensemble DCN ranks first overall at 71.6% exact match and 80.4% F1 on the test data.
The DCN has the capability to estimate the start and end points of the answer span multiple times, each time conditioned on its previous estimates. By doing so, the model is able to explore local maxima corresponding to multiple plausible answers, as is shown in Figure 5.
For example, Question 1 in Figure 5 demonstrates an instance where the model initially guesses an incorrect start point and a correct end point. In subsequent iterations, the model adjusts the start point, ultimately arriving at the correct start point in iteration 3. Similarly, the model gradually shifts probability mass for the end point to the correct word.
Question 2 shows an example in which both the start and end estimates are initially incorrect. The model then settles on the correct answer in the next iteration.
3https://rajpurkar.github.io/SQuAD-explorer
While the dynamic nature of the decoder allows the model to escape initial local maxima corresponding to incorrect answers, Question 3 demonstrates a case where the model is unable to decide between multiple local maxima despite several iterations. Namely, the model alternates between the answers “charged particle beam” and “particle beam weapons” indefinitely. Empirically, we observe that the model, trained with a maximum iteration of 4, takes 2.7 iterations to converge to an answer on average.
Model Ablation The performance of our model and its ablations on the SQuAD development set is shown in Table 2. On the decoder side, we experiment with various pool sizes for the HMN maxout layers, using a 2-layer MLP instead of a HMN, and forcing the HMN decoder to a single iteration. Empirically, we achieve the best performance on the development set with an iterative HMN
with pool size 16, and find that the model consistently benefits from a deeper, iterative decoder network. The performance improves as the number of maximum allowed iterations increases, with little improvement after 4 iterations. On the encoder side, replacing the coattention mechanism with an attention mechanism similar to Wang & Jiang (2016b) by setting CD to QAD in equation 3 results in a 1.9 point F1 drop. This suggests that, at an additional cost of a softmax computation and a dot product, the coattention mechanism provides a simple and effective means to better encode the document and question sequences. Further studies, such as performance without attention and performance on questions requiring different types of reasoning can be found in the appendix.
Performance across length One point of interest is how the performance of the DCN varies with respect to the length of document. Intuitively, we expect the model performance to deteriorate with longer examples, as is the case with neural machine translation (Luong et al., 2015). However, as in shown in Figure 6, there is no notable performance degradation for longer documents and questions contrary to our expectations. This suggests that the coattentive encoder is largely agnostic to long documents, and is able to focus on small sections of relevant text while ignoring the rest of the (potentially very long) document. We do note a performance degradation with longer answers. However, this is intuitive given the nature of the evaluation metric. Namely, it becomes increas-
ingly challenging to compute the correct word span as the number of words increases.
Performance across question type Another natural way to analyze the performance of the model is to examine its performance across question types. In Figure 7, we note that the mean F1 of DCN exceeds those of previous systems (Wang & Jiang, 2016b; Yu et al., 2016) across all question types. The DCN, like other models, is adept at “when” questions and struggles with the more complex “why” questions.
Breakdown of F1 distribution Finally, we note that the DCN performance is highly bimodal. On the development set, the model perfectly predicts (100% F1) an answer for 62.2% of examples and predicts a completely wrong answer (0% F1) for 16.3% of examples. That is, the model picks out partial answers only 21.5% of the time. Upon qualitative inspections of the 0% F1 answers, some of which are shown in Appendix A.4, we observe that when the model is wrong, its mistakes tend to have the correct “answer type” (eg. person for a “who” question, method for a “how” question) and the answer boundaries encapsulate a well-defined phrase.
5 CONCLUSION
We proposed the Dynamic Coattention Network, an end-to-end neural network architecture for question answering. The DCN consists of a coattention encoder which learns co-dependent representations of the question and of the document, and a dynamic decoder which iteratively estimates the answer span. We showed that the iterative nature of the model allows it to recover from initial local maxima corresponding to incorrect predictions. On the SQuAD dataset, the DCN achieves the state of the art results at 75.9% F1 with a single model and 80.4% F1 with an ensemble. The DCN significantly outperforms all other models.
ACKNOWLEDGMENTS
We thank Kazuma Hashimoto and Bryan McCann for their help and insights.
A APPENDIX
A.1 PERFORMANCE WITHOUT ATTENTION
In our experiments, we also investigate a model without any attention mechanism. In this model, the encoder is a simple LSTM network that first ingests the question and then ingests the document. The hidden states corresponding to words in the document is then passed to the decoder. This model achieves 33.3% exact match and 41.9% F1, significantly worse than models with attention.
A.2 SAMPLES REQUIRING DIFFERENT TYPES OF REASONING
We generate predictions for examples requiring different types of reasoning, given by Rajpurkar et al. (2016). Because this set of examples is very limited, they do not conclusively demonstrate the effectiveness of the model on different types of reasoning tasks. Nevertheless, these examples show that the DCN is a promising architecture for challenging question answering tasks including those that involve reasoning over multiple sentences.
WHAT IS THE RANKINE CYCLE SOMETIMES CALLED?
The Rankine cycle is sometimes referred to as a practical Carnot cycle because, when an efficient turbine is used, the TS diagram begins to resemble the Carnot cycle.
Type of reasoning Lexical variation (synonymy)
Ground truth practical Carnot cycle
Prediction practical Carnot cycle
WHICH TWO GOVERNING BODIES HAVE LEGISLATIVE VETO POWER?
While the Commision has a monopoly on initiating legislation, the European Parliament and the Council of the European Union have powers of amendment and veto during the legislative progress.
Type of reasoning Lexical variation (world knowledge)
Ground truth the European Parliament and the Council of the European Union
Prediction European Parliament and the Council of the European Union
WHAT SHAKESPEARE SCHOLAR IS CURRENTLY ON THE UNIVERSITYS FACULTY?
Current faculty include the anthropologist Marshall Sahlins, historian Dipesh Chakrabarty, ... Shakespeare scholar David Bevington, and renowned political scientists John Mearsheimer and Robert Pape.
Type of reasoning Syntactic variation
Ground truth David Bevington
Prediction David Bevington
WHAT COLLECTION DOES THE V&A THEATRE & PERFORMANCE GALLERIES HOLD?
The V&A Theatre & Performance galleries, formerly the Theatre Museum, opened in March 2009. The collections are stored by the V&A, and are available for research, exhibitions and other shows. They hold the UK’s biggest national collection of material about live performance in the UK since Shakespeare’s day, covering drama, dance, musical theatre, circus, music hall, rock and pop, and most other forms of live entertainment.
Type of reasoning Multiple sentence reasoning
Ground truth Material about live performance
Prediction UK’s biggest national collection of material about live performance in the UK since Shakespeare’s day
WHAT IS THE MAIN GOAL OF CRIMINAL PUNISHMENT OF CIVIL DISOBEDIENTS?
Type of reasoning Ambiguous
Along with giving the offender his ”just deserts”, achieving crime control via incapacitation and deterrence is a major goal of crime punishment.
Ground truth achieving crime control via incapacitation and deterrence
Prediction achieving crime control via incapacitation and deterrence
A.3 SAMPLES OF CORRECT SQUAD PREDICTIONS BY THE DYNAMIC COATTENTION NETWORK
HOW DID THE MONGOLS ACQUIRE CHINESE PRINTING TECHNOLOGY?
ID: 572882242ca10214002da420
The Mongol rulers patronized the Yuan printing industry. Chinese printing technology was transferred to the Mongols through Kingdom of Qocho and Tibetan intermediaries. Some Yuan documents such as Wang Zhen’s Nong Shu were printed with earthenware movable type, a technology invented in the 12th century. However, most published works were still produced through traditional block printing techniques. The publication of a Taoist text inscribed with the name of Tregene Khatun, gedei’s wife, is one of the first printed works sponsored by the Mongols. In 1273, the Mongols created the Imperial Library Directorate, a government-sponsored printing office. The Yuan government established centers for printing throughout China. Local schools and government agencies were funded to support the publishing of books.
Ground truth through Kingdom of Qocho and Tibetan intermediaries
Prediction: through Kingdom of Qocho and Tibetan intermediaries
WHO APPOINTS ELDERS?
ID 5730d473b7151e1900c0155b
Elders are called by God, affirmed by the church, and ordained by a bishop to a ministry of Word, Sacrament, Order and Service within the church. They may be appointed to the local church, or to other valid extension ministries of the church. Elders are given the authority to preach the Word of God, administer the sacraments of the church, to provide care and counseling, and to order the life of the church for ministry and mission. Elders may also be assigned as District Superintendents, and they are eligible for election to the episcopacy. Elders serve a term of 23 years as provisional Elders prior to their ordination.
Ground truth bishop, the local church
Prediction a bishop
AN ALGORITHM FOR X WHICH REDUCES TO C WOULD ALLOW US TO DO WHAT?
ID 56e1ce08e3433e14004231a6
This motivates the concept of a problem being hard for a complexity class. A problem X is hard for a class of problems C if every problem in C can be reduced to X. Thus no problem in C is harder than X, since an algorithm for X allows us to solve any problem in C. Of course, the notion of hard problems depends on the type of reduction being used. For complexity classes larger than P, polynomial-time reductions are commonly used. In particular, the set of problems that are hard for NP is the set of NP-hard problems.
Ground truth solve any problem in C
Prediction solve any problem in C
HOW MANY GENERAL QUESTIONS ARE AVAILABLE TO OPPOSITION LEADERS?
ID 572fd7b8947a6a140053cd3e
Parliamentary time is also set aside for question periods in the debating chamber. A ”General Question Time” takes place on a Thursday between 11:40 a.m. and 12 p.m. where members can direct questions to any member of the Scottish Government. At 2.30pm, a 40-minute long themed ”Question Time” takes place, where members can ask questions of ministers in departments that are selected for questioning that sitting day, such as health and justice or education and transport. Between 12 p.m. and 12:30 p.m. on Thursdays, when Parliament is sitting, First Minister’s Question Time takes place. This gives members an opportunity to question the First Minister directly on issues under their jurisdiction. Opposition leaders ask a general question of the First Minister and then supplementary questions. Such a practice enables a ”lead-in” to the questioner, who then uses their supplementary question to ask the First Minister any issue. The four general questions available to opposition leaders are:
Ground truth four
Prediction four
WHAT ARE SOME OF THE ACCEPTED GENERAL PRINCIPLES OF EUROPEAN UNION LAW?
ID 5726a00cf1498d1400e8e551
The principles of European Union law are rules of law which have been developed by the European Court of Justice that constitute unwritten rules which are not expressly provided for in the treaties but which affect how European Union law is interpreted and applies. In formulating these principles, the courts have drawn on a variety of sources, including: public international law and legal doctrines and principles present in the legal systems of European Union member states and in the jurisprudence of the European Court of Human Rights. Accepted general principles of European Union Law include fundamental rights (see human rights), proportionality, legal certainty, equality before the law and subsidiarity.
Ground truth fundamental rights (see human rights), proportionality, legal certainty, equality before the law and subsidiarity
Prediction fundamental rights (see human rights), proportionality, legal certainty, equality before the law and subsidiarity
WHY WAS TESLA RETURNED TO GOSPIC?
ID 56dfaa047aa994140058dfbd
On 24 March 1879, Tesla was returned to Gospi under police guard for not having a residence permit. On 17 April 1879, Milutin Tesla died at the age of 60 after contracting an unspecified illness (although some sources say that he died of a stroke). During that year, Tesla taught a large class of students in his old school, Higher Real Gymnasium, in Gospi.
Ground truth not having a residence permit
Prediction not having a residence permit
A.4 SAMPLES OF INCORRECT SQUAD PREDICTIONS BY THE DYNAMIC COATTENTION NETWORK
WHAT IS ONE SUPPLEMENTARY SOURCE OF EUROPEAN UNION LAW?
ID 5725c3a9ec44d21400f3d506
European Union law is applied by the courts of member states and the Court of Justice of the European Union. Where the laws of member states provide for lesser rights European Union law can be
enforced by the courts of member states. In case of European Union law which should have been transposed into the laws of member states, such as Directives, the European Commission can take proceedings against the member state under the Treaty on the Functioning of the European Union. The European Court of Justice is the highest court able to interpret European Union law. Supplementary sources of European Union law include case law by the Court of Justice, international law and general principles of European Union law.
Ground truth international law
Prediction case law by the Court of Justice
Comment The prediction produced by the model is correct, however it was not selected by Mechanical Turk annotators.
WHO DESIGNED THE ILLUMINATION SYSTEMS THAT TESLA ELECTRIC LIGHT &
MANUFACTURING INSTALLED?
ID 56e0d6cf231d4119001ac424
After leaving Edison’s company Tesla partnered with two businessmen in 1886, Robert Lane and Benjamin Vail, who agreed to finance an electric lighting company in Tesla’s name, Tesla Electric Light & Manufacturing. The company installed electrical arc light based illumination systems designed by Tesla and also had designs for dynamo electric machine commutators, the first patents issued to Tesla in the US.
Ground truth Tesla
Prediction Robert Lane and Benjamin Vail
Comment The model produces an incorrect prediction that corresponds to people that funded Tesla, instead of Tesla who actually designed the illumination system. Empirically, we find that most mistakes made by the model have the correct type (eg. named entity type) despite not including types as prior knowledge to the model. In this case, the incorrect response has the correct type of person.
CYDIPPID ARE TYPICALLY WHAT SHAPE?
ID 57265746dd62a815002e821a
Cydippid ctenophores have bodies that are more or less rounded, sometimes nearly spherical and other times more cylindrical or egg-shaped; the common coastal ”sea gooseberry,” Pleurobrachia, sometimes has an egg-shaped body with the mouth at the narrow end, although some individuals are more uniformly round. From opposite sides of the body extends a pair of long, slender tentacles, each housed in a sheath into which it can be withdrawn. Some species of cydippids have bodies that are flattened to various extents, so that they are wider in the plane of the tentacles.
Ground truth more or less rounded, egg-shaped
Prediction spherical
Comment Although the mistake is subtle, the prediction is incorrect. The statement “are more or less rounded, sometimes nearly spherical” suggests that the entity is more often “rounded” than “spherical” or “cylindrical” or “egg-shaped” (an answer given by an annotator). This suggests that the model has trouble discerning among multiple intuitive answers due to a lack of understanding of the relative severity of “more or less” versus “sometimes” and “other times”.

Summary: The paper proposes a novel deep neural network architecture for the task of question answering on the SQuAD dataset. The model consists of two main components -- coattention encoder and dynamic pointer decoder. The encoder produces attention over the question as well as over the document in parallel and thus learns co-dependent representations of the question and the document. The decoder predicts the starting and the end token of the answer iteratively, with the motivation that multiple iterations will help the model escape local maxima and thus will reduce the errors made by the model. The proposed model achieved state-of-art result on SQuAD dataset at the time of writing the paper. The paper reports some analyses of the results such as performance across question types, document, question, answer lengths, etc. The paper also performs some ablation studies such as performing only single round of iteration on decoder, etc.
Strengths:
1. The paper is well-motivated with two main motivations -- co-attending to the document and the question, and iteratively producing the answer.
2. The proposed model architecture is novel and the design choices made seem reasonable.
3. The experiments show that the proposed model outperforms the existing model (at the time of writing the paper) on the SQuAD dataset by significant margin.
4. The analyses of the results and the ablation studies performed (as per someone's request) provide insights into the various modelling design choices made.
Weaknesses/Questions/Suggestions:
1. In order to gain insights into how much each additional iteration in the decoder help, I would like to see the following -- for every iteration report the mean F1 for questions that converged in that iteration along with the number of questions that converged in that iteration.
2. Example of Question 3 in figure 5 is an interesting example where the model is unable to decide between multiple local maxima despite several iterations. Could authors please report how often this happens?
3. In order to estimate how much modelling of attention in the encoder helps, it would be good if authors could report the performance of the model when attention is not modeled at all in the encoder (neither over question, nor over document).
4. I would like to see the variation in the performance of the proposed model for questions that require different types of reasoning (table 3 in SQuAD paper). This would provide insights into what are the strengths and weaknesses of the proposed model w.r.t the type reasoning required.
5. In Wang and Jiang (2016), the attention is predicted over question for each word in the document. But in table 2, when performing ablation study to make the proposed model similar to Wang and Jiang, C^D is set to C^Q. But isn’t C^Q attention over document for each word in the question? So, how is this similar to Wang and Jiang’s attention? I think QA^D will be similar to Wang and Jiang's attention since QA^D is attention over question for each word in the document. Please clarify.
6. In section 2.1, “n” and “m” are swapped when explaining the Document and Question encoding matrix. Please fix it.
Review Summary: The paper presents a novel and interesting model for the task of question answering on SQuAD dataset and shows that the model outperforms existing models. However, to gain more insights into the functioning of the model, I would like see more analyses of the results and one more ablation study (see weaknesses section above).


Gradients have been used to quantify feature importance in machine learning models. Unfortunately, in nonlinear deep networks, not only individual neurons but also the whole network can saturate, and as a result an important input feature can have a tiny gradient. We study various networks, and observe that this phenomena is indeed widespread, across many inputs. We propose to examine interior gradients, which are gradients of counterfactual inputs constructed by scaling down the original input. We apply our method to the GoogleNet architecture for object recognition in images, as well as a ligand-based virtual screening network with categorical features and an LSTM based language model for the Penn Treebank dataset. We visualize how interior gradients better capture feature importance. Furthermore, interior gradients are applicable to a wide variety of deep networks, and have the attribution property that the feature importance scores sum to the the prediction score. Best of all, interior gradients can be computed just as easily as gradients. In contrast, previous methods are complex to implement, which hinders practical adoption.
Gradients have been used to quantify feature importance in machine learning models. Unfortunately, in nonlinear deep networks, not only individual neurons but also the whole network can saturate, and as a result an important input feature can have a tiny gradient. We study various networks, and observe that this phenomena is indeed widespread, across many inputs.
We propose to examine interior gradients, which are gradients of counterfactual inputs constructed by scaling down the original input. We apply our method to the GoogleNet architecture for object recognition in images, as well as a ligand-based virtual screening network with categorical features and an LSTM based language model for the Penn Treebank dataset. We visualize how interior gradients better capture feature importance. Furthermore, interior gradients are applicable to a wide variety of deep networks, and have the attribution property that the feature importance scores sum to the the prediction score.
Best of all, interior gradients can be computed just as easily as gradients. In contrast, previous methods are complex to implement, which hinders practical adoption.
1 INTRODUCTION
Practitioners of machine learning regularly inspect the coefficients of linear models as a measure of feature importance. This process allows them to understand and debug these models. The natural analog of these coefficients for deep models are the gradients of the prediction score with respect to the input. For linear models, the gradient of an input feature is equal to its coefficient. For deep nonlinear models, the gradient can be thought of as a local linear approximation (Simonyan et al. (2013)). Unfortunately, (see the next section), the network can saturate and as a result an important input feature can have a tiny gradient.
While there has been other work (see Section 2.10) to address this problem, these techniques involve instrumenting the network. This instrumentation currently involves significant developer effort because they are not primitive operations in standard machine learning libraries. Besides, these techniques are not simple to understand—they invert the operation of the network in different ways, and have their own peculiarities—for instance, the feature importances are not invariant over networks that compute the exact same function (see Figure 14).
In contrast, the method we propose builds on the very familiar, primitive concept of the gradient—all it involves is inspecting the gradients of a few carefully chosen counterfactual inputs that are scaled versions of the initial input. This allows anyone who knows how to extract gradients—presumably even novice practitioners that are not very familiar with the network’s implementation—to debug the network. Ultimately, this seems essential to ensuring that deep networks perform predictably when deployed.
2 OUR TECHNIQUE
2.1 GRADIENTS DO NOT REFLECT FEATURE IMPORTANCE
Let us start by investigating the performance of gradients as a measure of feature importance. We use an object recognition network built using the GoogleNet architecture (Szegedy et al. (2014)) as a running example; we refer to this network by its codename Inception. (We present applications of our techniques to other networks in Section 3.) The network has been trained on the ImageNet object recognition dataset (Russakovsky et al. (2015)). It is is 22 layers deep with a softmax layer on top for classifying images into one of the 1000 ImageNet object classes. The input to the network is a 224× 224 sized RGB image. Before evaluating the use of gradients for feature importance, we introduce some basic notation that is used throughout the paper.
We represent a 224× 224 sized RGB image as a vector in R224×224×3. Let IncpL : R224×224×3 → [0, 1] be the function represented by the Inception network that computes the softmax score for the object class labeled L. Let 5IncpL(img) be the gradients of IncpL at the input image img. Thus, the vector 5IncpL(img) is the same size as the image and lies in R224×224×3. As a shorthand, we write5IncpLi,j,c(img) for the gradient of a specific pixel (i, j) and color channel c ∈ {R,G,B}.
We compute the gradients of IncpL (with respect to the image) for the highest-scoring object class, and then aggregate the gradients5IncpL(img) along the color dimension to obtain pixel importance scores.1
∀i, j : PLi,j(img) ::= Σc∈{R,G,B}| 5 Incp L i,j,c(img)| (1)
Next, we visualize pixel importance scores by scaling the intensities of the pixels in the original image in proportion to their respective scores; thus, higher the score brighter would be the pixel. Figure 1a shows a visualization for an image for which the highest scoring object class is “reflex camera” with a softmax score of 0.9938.
1 These pixel importance scores are similar to the gradient-based saliency map defined by Simonyan et al. (2013) with the difference being in how the gradients are aggregated along the color channel.
Intuitively, one would expect the the high gradient pixels for this classification to be ones falling on the camera or those providing useful context for the classification (e.g., the lens cap). However, most of the highlighted pixels seem to be on the left or above the camera, which to a human seem not essential to the prediction. This could either mean that (1) the highlighted pixels are somehow important for the internal computation performed by the Inception network, or (2) gradients of the image fail to appropriately quantify pixel importance.
Let us consider hypothesis (1). In order to test it we ablate parts of the image on the left and above the camera (by zeroing out the pixel intensities) and run the ablated image through the Inception network. See Figure 1b. The top predicted category still remains “reflex camera” with a softmax score of 0.9966 — slightly higher than before. This indicates that the ablated portions are indeed irrelevant to the classification. On computing gradients of the ablated image, we still find that most of the high gradient pixels lie outside of the camera. This suggests that for this image, it is in fact hypothesis (2) that holds true. Upon studying more images (see Figure 4), we find that the gradients often fail to highlight the relevant pixels for the predicted object label.
2.2 SATURATION
In theory, it is easy to see that the gradients may not reflect feature importance if the prediction function flattens in the vicinity of the input, or equivalently, the gradient of the prediction function with respect to the input is tiny in the vicinity of the input vector. This is what we call saturation, which has also been reported in previous work (Shrikumar et al. (2016), Glorot & Bengio (2010)).
We analyze how widespread saturation is in the Inception network by inspecting the behavior of the network on counterfactual images obtained by uniformly scaling pixel intensities from zero to their values in an actual image. Formally, given an input image img ∈ R224×224×3, the set of counterfactual images is
{α img | 0 ≤ α ≤ 1} (2)
Figure 2a shows the trend in the softmax output of the highest scoring class, for thirty randomly chosen images form the ImageNet dataset. More specifically, for each image img, it shows the trend in IncpL(α img) as α varies from zero to one with L being the label of highest scoring object class for img. It is easy to see that the trend flattens (saturates) for all images α increases. Notice that saturation is present even for images whose final score is significantly below 1.0. Moreover, for a majority of images, saturation happens quite soon when α = 0.2.
One may argue that since the output of the Inception network is the result of applying the softmax function to a vector of activation values, the saturation is expected due to the squashing property of the softmax function. However, as shown in Figure 2b, we find that even the pre-softmax activation scores for the highest scoring class saturate.
In fact, to our surprise, we found that the saturation is inherently present in the Inception network and the outputs of the intermediate layers also saturate. We plot the distance between the intermediate layer neuron activations for a scaled down input image and the actual input image with respect to the scaling parameter, and find that the trend flattens. Due to lack of space, we provide these plots in Figure 12 in the appendix.
It is quite clear from these plots that saturation is widespread across images in the Inception network, and there is a lot more activity in the network for counterfactual images at relatively low values of the scaling parameter α. This observation forms the basis of our technique for quantifying feature importance.
Note that it is well known that the saturation of gradients prevent the model from converging to a good quality minima (Glorot & Bengio (2010)). So one may expect good quality models to not have saturation and hence for the (final) gradients to convey feature importance. Clearly, our observations on the Inception model show that this is not the case. It has good prediction accuracy, but also exhibits saturation (see Figure 2). Our hypothesis is that the gradients of important features are not saturated early in the training process. The gradients only saturate after the features have been learned adequately, i.e., the input is far away from the decision boundary.
2.3 INTERIOR GRADIENTS
We study the importance of input features in a prediction made for an input by examining the gradients of the counterfactuals obtained by scaling the input; we call this set of gradients interior gradients.
While the method of examining gradients of counterfactual inputs is broadly applicable to a wide range of networks, we first explain it in the context of Inception. Here, the counterfactual image inputs we consider are obtained by uniformly scaling pixel intensities from zero to their values in the actual image (this is the same set of counterfactuals that was used to study saturation). The interior gradients are the gradients of these images.
InteriorGrads(img) ::= {5Incp(α img) | 0 ≤ α ≤ 1} (3)
These interior gradients explore the behavior of the network along the entire scaling curve depicted in Figure 2a, rather than at a specific point. We can aggregate the interior gradients along the color dimension to obtain interior pixel importance scores using equation 1.
InteriorPixelImportance(img) ::= {P(α img) | 0 ≤ α ≤ 1} (4)
We individually visualize the pixel importance scores for each scaling parameter α by scaling the intensities of the pixels in the actual image in proportion to their scores. The visualizations show how the importance of each pixel evolves as we scale the image, with the last visualization being identical to one generated by gradients at the actual image. In this regard, the interior gradients offer strictly more insight into pixel importance than just the gradients at the actual image.
Figure 3 shows the visualizations for the “reflex camera” image from Figure 1a for various values of the scaling parameter α. The plot in the top right corner shows the trend in the absolute magnitude of the average pixel importance score. The magnitude is significantly larger at lower values of α and nearly zero at higher values — the latter is a consequence of saturation. Note that each visualization is only indicative of the relative distribution of the importance scores across pixels and not the absolute magnitude of the scores, i.e., the later snapshots are responsible for tiny increases in the scores as the chart in the top right depicts.
The visualizations show that at lower values of α, the pixels that lie on the camera are most important, and as α increases, the region above the camera gains importance. Given the high magnitude of gradients at lower values of α, we consider those gradients to be the primary drivers of the final prediction score. They are more indicative of feature importance in the prediction compared to the gradients at the actual image (i.e., when α = 1).
The visualizations of the interior pixel gradients can also be viewed together as a single animation that chains the visualizations in sequence of the scaling parameter. This animation offers a concise yet complete summary of how pixel importance moves around the image as the scaling parameter increase from zero to one.
Rationale. While measuring saturation via counterfactuals seems natural, using them for quantifying feature importance deserves some discussion. The first thing one may try to identify feature importance is to examine the deep network like one would with human authored code. This seems hard; just as deep networks employ distributed representations (such as embeddings), they perform convoluted (pun intended) distributed reasoning. So instead, we choose to probe the network with several counterfactual inputs (related to the input at hand), hoping to trigger all the internal workings of the network. This process would help summarize the effect of the network on the protagonist input; the assumption being that the input is human understandable. Naturally, it helps to work with gradients in this process as via back propagation, they induce an aggregate view over the function computed by the neurons.
Interior gradients use counterfactual inputs to artifactually induce a procedure on how the networks attention moves across the image as it compute the final prediction score. From the animation, we gather that the network focuses on strong and distinctive patterns in the image at lower values of the scaling parameter, and subtle and weak patterns in the image at higher values. Thus, we speculate that the network’s computation can be loosely abstracted by a procedure that first recognize distinctive features of the image to make an initial prediction, and then fine tunes (these are small score jumps as the chart in Figure 3 shows) the prediction using weaker patterns in the image.
2.4 CUMULATING INTERIOR GRADIENTS
A different summarization of the interior gradients can be obtained by cumulating them. While there are a few ways of cumulating counterfactual gradients, the approach we take has the nice attribution property (Proposition 1) that the feature importance scores approximately add up to the prediction score. The feature importance scores are thus also referred to as attributions.
Notice that the set of counterfactual images {α img | 0 ≤ α ≤ 1} fall on a straight line path in R224×224×3. Interior gradients — which are the gradients of these counterfactual images — can be cumulated by integrating them along this line. We call the resulting gradients as integrated gradients. In what follows, we formalize integrated gradients for an arbitrary function F : Rn → [0, 1] (representing a deep network), and an arbitrary set of counterfactual inputs falling on a path in Rn.
Let x ∈ Rn be the input at hand, and γ = (γ1, . . . , γn) : [0, 1]→ Rn be a smooth function specifying the set of counterfactuals; here, γ(0) is the baseline input (for Inception, a black image), and γ(1) is the actual input (for Inception, the image being studied). Specifically, {γ(α) | 0 ≤ α ≤ 1} is the set of counterfactuals (for Inception, a series of images that interpolate between the black image and the actual input).
The integrated gradient along the ith dimension for an input x ∈ Rn is defined as follows.
IntegratedGradsi(x) ::= ∫ 1 α=0 ∂F (γ(α)) ∂γi(α) ∂γi(α) ∂α dα (5)
where ∂F (x)∂xi is the gradient of F along the i th dimension at x.
A nice technical property of the integrated gradients is that they add up to the difference between the output of F at the final counterfactual γ(1) and the baseline counterfactual γ(0). This is formalized by the proposition below, which is an instantiation of the fundamental theorem of calculus for path integrals.
Proposition 1 If F : Rn → R is differentiable almost everywhere 2, and γ : [0, 1] → Rn is smooth then
Σni=1IntegratedGradsi(x) = F (γ(1))− F (γ(0))
For most deep networks, it is possible to choose counterfactuals such that the prediction at the baseline counterfactual is near zero (F (γ(0)) ≈ 0).3 For instance, for the Inception network, the counterfactual defined by the scaling path satisfies this property as Incp(0224×224×3) ≈ 0. In such cases, it follows from the Proposition that the integrated gradients form an attribution of the prediction output F (x), i.e., they almost exactly distribute the output to the individual input features.
The additivity property provides a form of sanity checking for the integrated gradients and ensures that we do not under or over attribute to features. This is a common pitfall for attribution schemes based on feature ablations, wherein, an ablation may lead to small or a large change in the prediction score depending on whether the ablated feature interacts disjunctively or conjunctively to the rest of the features. This additivity is even more desirable when the networks score is numerically critical, i.e., the score is not used purely in an ordinal sense. In this case, the attributions (together with additivity) guarantee that the attributions are in the units of the score, and account for all of the score.
We note that these path integrals of gradients have been used to perform attribution in the context of small non-linear polynomials (Sun & Sundararajan (2011)), and also within the cost-sharing literature in economics where function at hand is a cost function that models the cost of a project as a function of the demands of various participants, and the attributions correspond to cost-shares. The specific path we use corresponds to a cost-sharing method called Aumann-Shapley (Aumann & Shapley (1974)).
Computing integrated gradients. The integrated gradients can be efficiently approximated by Riemann sum, wherein, we simply sum the gradients at points occurring at sufficiently small intervals along the path of counterfactuals.
IntegratedGradsapproxi (x) ::= Σ m k=1 ∂F (γ(k/m)) ∂γi(α) (γ( km )− γ( k−1 m )) (6)
Here m is the number of steps in the Riemman approximation of the integral. Notice that the approximation simply involves computing the gradient in a for loop; computing the gradient is central to deep learning and is a pretty efficient operation. The implementation should therefore
2Formally, this means that the partial derivative of F along each input dimension satisfies Lebesgue’s integrability condition, i.e., the set of discontinuous points has measure zero. Deep networks built out of Sigmoids, ReLUs, and pooling operators should satisfy this condition.
3We did have trouble finding a baseline couterfactual for an RNN model that simulated the workings of a traffic light intersection between a main road and a side street; the naive benchmark counterfactual was one of no traffic at either intersection. But this did not have the lack of semantics that a black image or pure noise has for the Inception network. While no interesting labels are activated for the black image supplied to the Inception network, the same is not true for the “no traffic” benchmark supplied to the RNN model.
be straightforward in most deep learning frameworks. For instance, in TensorFlow (ten), it essentially amounts to calling tf.gradients in a loop over the set of counterfactual inputs (i.e., γ( km ) for k = 1, . . . ,m), which could also be batched. Going forward, we abuse the term “integrated gradients” to refer to the approximation described above.
Integrated gradients for Inception. We compute the integrated gradients for the Inception network using the counterfactuals obtained by scaling the input image; γ(α) = α img where img is the input image. Similar to the interior gradients, the integrated gradients can also be aggregated along the color channel to obtain pixel importance scores which can then be visualized as discussed earlier. Figure 4 shows these visualizations for a bunch of images. For comparison, it also presents the corresponding visualization obtained from the gradients at the actual image. From the visualizations, it seems quite evident that the integrated gradients are better at capturing important features.
2.5 TWO NORMATIVE AXIOMS FOR FEATURE ATTRIBUTIONS
We discuss two desirable axioms for feature attribution methods. We show that our integrated gradients method satisfies both. On the other hand, the other feature attribution methods in literature break one of the two axioms. These methods include DeepLift (Shrikumar et al. (2016)), Layer-wise relevance propagation (LRP) (Binder et al. (2016)), Deconvolutional networks (Zeiler & Fergus (2014)), and Guided back-propagation (Springenberg et al. (2014)).
Sensitivity.
A highly desirable property for feature attributions is Sensitivity. If a non-zero change in a single input variable (holding all other variables fixed) changes the output by a non-zero amount, then this variable should be given a non-zero attribution. In other words, attribution should be sensitive to change.
Integrated Gradients (ignoring the approximation in computing integrals) satisfies Sensitivity. The attribution to the variable is in fact equal to the change in function value (this is a one-variable instance of Proposition 1).
While this property is satisfied by the integrated gradients method, it is broken by Gradients, Deconvolution networks (DeConvNets) (Zeiler & Fergus (2014)), Guided back-propagation (Springenberg et al. (2014)).
Gradients break Sensitivity due to saturation (see Section 2.2), i.e., the prediction function may flatten at the input and thus have zero gradient despite the function value at the input being different from that at the benchmark. For a concrete example, consider a one variable, one ReLU network, f(x) = 1−ReLU(1−x). Suppose we change the input from x = 0 to x = 2. The function changes from 0 to 1, but because f is flat at x = 1, the gradient method gives attribution of 0 to x, violating sensitivity. We defer the counterexamples for other methods to Appendix B.
Implementation Invariance. Two networks can be functionally equivalent, i.e., their outputs are equal for all inputs, despite having very different implementations. We would like our attribution method to satisfy Implementation Invariance, i.e., the attributions are always identical for two functionally equivalent networks. To motivate this, notice that attribution can be colloquially defined as distributing the blame (or credit) for the output to the input features. Such a definition does not refer to implementation details. Moreover, the common practice of machine learning tends to evaluate the models from an input-output point of view, where implementations are purely means to an end.
Attributions generated by integrated gradients (or gradients, or any function of the interior gradients) satisfy Implementation Invariance since they are based only on the gradients of the function represented by the network. On the other hand, this fundamental property is unfortunately broken for the DeepLift and LRP methods. Below, we describe intuition for why Implementation Invariance is broken by these methods; a concrete example is provided in Appendix B.
First, notice that gradients are invariant to implementation. In fact, the chain-rule for gradients ∂f ∂g = ∂f ∂h · ∂h ∂g is essentially about implementation invariance. To see this, think of g and f as the input and output of a system. The gradient of output f to input g can be computed either directly by
∂f ∂g , ignoring the intermediate function h, or by invoking the chain rule via h. This is exactly how backpropagation works.
As previously discussed, gradients don’t satisfy sensitivity, and are therefore unsuitable for attribution. Methods like DeepLift tackle this issue by introducing a benchmark, and in some sense try to compute “discrete gradients” instead of gradients. They use a backpropagation procedure for composing discrete gradients. Unfortunately, such approaches are problematic because chain rule does not hold for discrete gradients in general. Formally f(x1)−f(x0)g(x1)−g(x0) = f(x1)−f(x0) h(x1)−h(x0) · h(x1)−h(x0) g(x1)−g(x0) does not hold, and therefore these methods fail to satisfy implementation invariance.
If an attribution method fails to satisfy Implementation Invariance, the attributions are potentially sensitive to unimportant aspects of the models. For instance, in the example in Section B, the network architecture has more degrees of freedom than needed for representing the function, and as a result there are two set of values for the network parameters that lead to the same function. The training procedure can converge at either set of values depending on the initializtion or for other reasons, but the underlying network function would remain the same. It is undesirable that attributions differ for such reasons.
2.6 A FULL AXIOMATIZATION
There are many methods that satisfy Implementation Invariance and Sensitivity. In this section we show that Integrated Gradients is not just one of them. It is in fact also the only method that satisfies an extended set of axioms. The additional axioms are reasonably natural but perhaps not as fundamental to attribution. As we shall see in the next section there does not seem to be a perfect empirical evaluation for attribution methods. We hope that these axioms provide a theoretical framework for evaluating attribution methods, which provide a good complement to empirical evaluations.
As discussed earlier Integrated Gradients corresponds to a method called Aumann-Shapley studied by economists in the context of cost-sharing. (The function at hand is a cost-function whose input variables are demands of different participants and attributions correspond to cost-shares.) Here is the list of axioms, borrowed from the cost-sharing literature Billera & Heath (1982); a longer discussion of the desirability of these axioms in the context of attribution can be found in Sun & Sundararajan (2011).
• Dummy: If the function implemented by the deep network does not depend on a variable, then the attribution to it is always zero.
• Additivity: For all inputs, the attributions for a function f1+f2 is the sum of the attributions for the function f1 and the function f2.
• Completeness: The attributions add up to the difference between the function values at the input and the benchmark.
• Scale Invariance: Informally, if the inputs to two networks differ in the scale of one of the variables (say Farenheit and Celsius), but have the same output for corresponding (rescaled) inputs, then the attributions should be identical.
• Proportional Attributions for Homogenous Variables: If a function can be represented by the sum of the two variables, then the two variables should receive attributions proportional to their input values.
Proposition 2 Billera & Heath (1982) Integrated Gradients is the unique method that satisfies all of the axioms above.
2.7 AN EMPIRICAL EVALUATION OF OUR APPROACH
We now discuss an emprical evaluation of integrated gradients as a measure of feature importance, using gradients as a benchmark.
Pixel ablations. The first evaluation is based on a method by Samek et al. (2015). Here we ablate4 the top 5000 pixels (10% of the image) by importance score, and compute the score drop for the highest scoring object class. The ablation is performed 100 pixels at a time, in a sequence of 50 steps. At each perturbation step k we measure the average drop in score up to step k. This quantity is referred to a area over the perturbation curve (AOPC) by Samek et al. (2015).
Figure 5 shows the AOPC curve with respect to the number of perturbation steps for integrated gradients and gradients at the image. AOPC values at each step represent the average over a dataset of 150 randomly chosen images. It is clear that ablating the top pixels identified by integrated gradients leads to a larger score drop that those identified by gradients at the image.
Having said that, we note an important issue with the technique. The images resulting from pixel perturbation are often unnatural, and it could be that the scores drop simply because the network has never seen anything like it in training.
Localization. The second evaluation is to consider images with human-drawn bounding boxes around objects, and compute the percentage of pixel attribution inside the bounding box. We use the 2012 ImageNet object localization challenge dataset to get a set of human-drawn bounding boxes. We run our evaluation on 100 randomly chosen images satisfying the following properties — (1) the total size of the bounding box(es) is less than two thirds of the image size, and (2) ablating the bounding box significantly drops the prediction score for the object class. (1) is for ensuring that the boxes are not so large that the bulk of the attribution falls inside them by definition, and (2) is for ensuring that the boxed part of the image is indeed responsible for the prediction score for the image. We find that on 82 images the integrated gradients technique leads to a higher fraction of the pixel attribution inside the box than gradients at the actual image. The average difference in the percentage pixel attribution inside the box for the two techniques is 8.4%.
While these results are promising, we note the following caveat. Integrated gradients are meant to capture pixel importance with respect to the prediction task. While for most objects, one would expect the pixel located on the object to be most important for the prediction, in some cases the context in which the object occurs may also contribute to the prediction. The cabbage butterfly image from Figure 4 is a good example of this where the pixels on the leaf are also surfaced by the integrated gradients.
Eyeballing. Ultimately, it was hard to come up with a perfect evaluation technique. So we did spend a large amount of time applying and eyeballing the results of our technique to various networks— the ones presented in this paper, as well as some networks used within products. For the Inception network, we welcome you to eyeball more visualizations in Figure 11 in the appendix and also at: https://github.com/ankurtaly/Attributions. While we found our method to beat gradients at the image for the most part, this is clearly a subjective process prone to interpretation and cherry-picking, but is also ultimately the measure of the utility of the approach—debugging inherently involves the human.
Finally, also note that we did not compare against other whitebox attribution techniques (e.g., DeepLift (Shrikumar et al. (2016))), because our focus was on black-box techniques that are easy to implement, so comparing against gradients seems like a fair comparison.
2.8 DEBUGGING NETWORKS
Despite the widespread application of deep neural networks to problems in science and technology, their internal workings largely remain a black box. As a result, humans have a limited ability to understand the predictions made by these networks. This is viewed as hindrance in scenarios where the bar for precision is high, e.g., medical diagnosis, obstacle detection for robots, etc. (dar (2016)). Quantifying feature importance for individual predictions is a first step towards understanding the behavior of the network; at the very least, it helps debug misclassified inputs, and sanity check the internal workings. We present evidence to support this below.
4Ablation in our setting amounts to zeroing out (or blacking out) the intensity for the R, G, B channels. We view this as a natural mechanism for removing the information carried by the pixel (than, say, randomizing the pixel’s intensity as proposed by Samek et al. (2015), especially since the black image is a natural baseline for vision tasks.
We use feature importance to debug misclassifications made by the Inception network. In particular, we consider images from the ImageNet dataset where the groundtruth label for the image not in the top five labels predicted by the Inception network. We use interior gradients to compute pixel importance scores for both the Inception label and the groundtruth label, and visualize them to gain insight into the cause for misclassification.
Figure 6 shows the visualizations for two misclassified images. The top image genuinely has two objects, one corresponding to the groundtruth label and other corresponding to the Inception label. We find that the interior gradients for each label are able to emphasize the corresponding objects. Therefore, we suspect that the misclassification is in the ranking logic for the labels rather than the recognition logic for each label. For the bottom image, we observe that the interior gradients are largely similar. Moreover, the cricket gets emphasized by the interior gradients for the mantis (Inception label). Thus, we suspect this to be a more serious misclassification, stemming from the recognition logic for the mantis.
2.9 DISCUSSION
Faithfullness. A natural question is to ask why gradients of counterfactuals obtained by scaling the input capture feature importance for the original image. First, from studying the visualizations in Figure 4, the results look reasonable in that the highlighted pixels capture features representative of the predicted class as a human would perceive them. Second, we confirmed that the network too seems to find these features representative by performing ablations. It is somewhat natural to expect that the Inception network is robust to to changes in input intensity; presumably there are some low brightness images in the training set.
However, these counterfactuals seem reasonable even for networks where such scaling does not correspond to a natural concept like intensity, and when the counterfactuals fall outside the training set; for instance in the case of the ligand-based virtual screening network (see Section 3.1). We speculate that the reason why these counterfactuals make sense is because the network is built by composing ReLUs. As one scales the input starting from a suitable baseline, various neurons activate, and the scaling process that does a somewhat thorough job of exploring all these events that contribute to the prediction for the input. There is an analogous argument for other operator such as max pool, average pool, and softmax—here the triggering events arent discrete but the argument is analogous.
Limitations of Approach. We discuss some limitations of our technique; in a sense these are limitations of the problem statement and apply equally to other techniques that attribute to base input features.
• Inability to capture Feature interactions: The models could perform logic that effectively combines features via a conjunction or an implication-like operations; for instance, it could be that a molecule binds to a site if it has a certain structure that is essentially a conjunction of certain atoms and certain bonds between them. Attributions or importance scores have no way to represent these interactions.
• Feature correlations: Feature correlations are a bane to the understandability of all machine learning models. If there are two features that frequently co-occur, the model is free to assign weight to either or both features. The attributions would then respect this weight assignment. But, it could be that the specific weight assignment chosen by the model is not human-intelligible. Though there have been approaches to feature selection that reduce feature correlations (Yu & Liu (2003)), it is unclear how they apply to deep models on dense input.
2.10 RELATED WORK
Over the last few years, there has been a vast amount work on demystifying the inner workings of deep networks. Most of this work has been on networks trained on computer vision tasks, and deals with understanding what a specific neuron computes (Erhan et al. (2009); Le (2013)) and interpreting the representations captured by neurons during a prediction (Mahendran & Vedaldi (2015); Dosovitskiy & Brox (2015); Yosinski et al. (2015)).
Our work instead focuses on understanding the network’s behavior on a specific input in terms of the base level input features. Our technique quantifies the importance of each feature in the prediction. Known approaches for accomplishing this can be divided into three categories.
Gradient based methods. The first approach is to use gradients of the input features to quantify feature importance (Baehrens et al. (2010); Simonyan et al. (2013)). This approach is the easiest to implement. However, as discussed earlier, naively using the gradients at the actual input does not accurate quantify feature importance as gradients suffer from saturation.
Score back-propagation based methods. The second set of approaches involve back-propagating the final prediction score through each layer of the network down to the individual features. These include DeepLift (Shrikumar et al. (2016)), Layer-wise relevance propagation (LRP) (Binder et al. (2016)), Deconvolutional networks (DeConvNets) (Zeiler & Fergus (2014)), and Guided backpropagation (Springenberg et al. (2014)). These methods largely differ in the backpropagation logic for various non-linear activation functions. While DeConvNets, Guided back-propagation and LRP rely on the local gradients at each non-linear activation function, DeepLift relies on the deviation in the neuron’s activation from a certain baseline input.
Similar to integrated gradients, the DeepLift and LRP also result in an exact distribution of the prediction score to the input features. However, as shown by Figure 14, the attributions are not invariant across functionally equivalent networks. Besides, the primary advantage of our method over all these methods is its ease of implementation. The aforesaid methods require knowledge of the network architecture and the internal neuron activations for the input, and involve implementing a somewhat complicated back-propagation logic. On the other hand, our method is agnostic to the network architectures and relies only on computing gradients which can done efficiently in most deep learning frameworks.
Model approximation based methods. The third approach, proposed first by Ribeiro et al. (2016a;b), is to locally approximate the behavior of the network in the vicinity of the input being explained with a simpler, more interpretable model. An appealing aspect of this approach is that it is completely agnostic to the structure of the network and only deals with its input-output behavior. The approximation is learned by sampling the network’s output in the vicinity of the input at hand. In this sense, it is similar to our approach of using counterfactuals. Since the counterfactuals are chosen somewhat arbitrarily, and the approximation is based purely on the network’s output at the counterfactuals, the faithfullness question is far more crucial in this setting. The method is also expensive to implement as it requires training a new model locally around the input being explained.
3 APPLICATIONS TO OTHER NETWORKS
The technique of quantifying feature importance by inspecting gradients of counterfactual inputs is generally applicable across deep networks. While for networks performing vision tasks, the counterfactual inputs are obtained by scaling pixel intensities, for other networks they may be obtained by scaling an embedding representation of the input.
As a proof of concept, we apply the technique to the molecular graph convolutions network of Kearnes et al. (2016) for ligand-based virtual screening and an LSTM model (Zaremba et al. (2014)) for the language modeling of the Penn Treebank dataset (Marcus et al. (1993)).
3.1 LIGAND-BASED VIRTUAL SCREENING
The Ligand-Based Virtual Screening problem is to predict whether an input molecule is active against a certain target (e.g., protein or enzyme). The process is meant to aid the discovery of new drug molecules. Deep networks built using molecular graph convolutions have recently been proposed by Kearnes et al. (2016) for solving this problem.
Once a molecule has been identified as active against a target, the next step for medicinal chemists is to identify the molecular features—formally, pharmacophores5—that are responsible for the ac-
5A pharmacophore is the ensemble of steric and electronic features that is necessary to ensure the a molecule is active against a specific biological target to trigger (or to block) its biological response.
tivity. This is akin to quantifying feature importance, and can be achieved using the method of integrated gradients. The attributions obtained from the method help with identifying the dominant molecular features, and also help sanity check the behavior of the network by shedding light on its inner workings. With regard to the latter, we discuss an anecdote later in this section on how attributions surfaced an anomaly in W1N2 network architecture proposed by Kearnes et al. (2016).
Defining the counterfactual inputs. The first step in computing integrated gradients is to define the set of counterfactual inputs. The network requires an input molecule to be encoded by hand as a set of atom and atom-pair features describing the molecule as an undirected graph. Atoms are featurized using a one-hot encoding specifying the atom type (e.g., C, O, S, etc.), and atom-pairs are featurized by specifying either the type of bond (e.g., single, double, triple, etc.) between the atoms, or the graph distance between them 6
The counterfactual inputs are obtained by scaling down the molecule features down to zero vectors, i.e., the set {αFeatures(mol) | 0 ≤ α ≤ 1} where Features(mol) is an encoding of the molecule into atom and atom-pair features.
The careful reader might notice that these counterfactual inputs are not valid featurizations of molecules. However, we argue that they are still valid inputs for the network. First, all operators in the network (e.g., ReLUs, Linear filters, etc.) treat their inputs as continuous real numbers rather than discrete zeros and ones. Second, all fields of the counterfactual inputs are bounded between zero and one, therefore, we don’t expect them to appear spurious to the network. We discuss this further in section 2.9
6This featurization is referred to as “simple” input featurization in Kearnes et al. (2016).
In what follows, we discuss the behavior of a network based on the W2N2-simple architecture proposed by Kearnes et al. (2016). On inspecting the behavior of the network over counterfactual inputs, we observe saturation here as well. Figure 13a shows the trend in the softmax score for the task PCBA-588342 for twenty five active molecules as we vary the scaling parameter α from zero to one. While the overall saturated region is small, saturation does exist near vicinity of the input (0.9 ≤ α ≤ 1). Figure 13b in the appendix shows that the total feature gradient varies significantly along the scaling path; thus, just the gradients at the molecule is fully indicative of the behavior of the network.
Visualizing integrated gradients. We cumulate the gradients of these counterfactual inputs to obtain an attribution of the prediction score to each atom and atom-pair feature. Unlike image inputs, which have dense features, the set of input features for molecules are sparse. Consequently, the attributions are sparse and can be inspected directly. Figure 7 shows heatmaps for the atom and atom-pair attributions for a specific molecule.
Using the attributions, one can easily identify the atoms and atom-pairs that that have a strongly positive or strongly negative contribution. Since the attributions add up to the final prediction score (see Proposition 1), the attribution magnitudes can be use for accounting the contributions of each feature. For instance, the atom-pairs that have a bond between them contribute cumulatively contribute 46% of the prediction score, while all other atom pairs cumulatively contribute −3%. We presented the attributions for 100 molecules active against a specific task to a few chemists. The chemists were able to immediately spot dominant functional groups (e.g., aromatic rings) being surfaced by the attributions. A next step could be cluster the aggregate the attributions across a large set of molecules active against a specific task to identify a common denominator of features shared by all active molecules.
Identifying Dead Features. We now discuss how attributions helped us spot an anomaly in the W1N2 architecture. On applying the integrated gradients method to the W1N2 network, we found that several atoms in the same molecule received the exact same attribution. For instance, for the molecule in Figure 7, we found that several carbon atoms at positions 2, 3, 14, 15, and 16 received the same attribution of 0.0043 despite being bonded to different atoms, for e.g., Carbon at position 3 is bonded to an Oxygen whereas Carbon at position 2 is not. This is surprising as one would expect two atoms with different neighborhoods to be treated differently by the network.
On investigating the problem further we found that since the W1N2 network had only one convolution layer, the atoms and atom-pair features were not fully convolved. This caused all atoms that
have the same atom type, and same number of bonds of each type to contribute identically to the network. This is not the case for networks that have two or more convolutional layers.
Despite the aforementioned problem, the W1N2 network had good predictive accuracy. One hypothesis for this is that the atom type and their neighborhoods are tightly correlated; for instance an outgoing double bond from a Carbon is always to another Carbon or Oxygen atom. As a result, given the atom type, an explicit encoding of the neighborhood is not needed by the network. This also suggests that equivalent predictive accuracy can be achieved using a simpler “bag of atoms” type model.
3.2 LANGUAGE MODELING
To apply our technique for language modeling, we study word-level language modeling of the Penn Treebank dataset (Marcus et al. (1993)), and apply an LSTM-based sequence model based on Zaremba et al. (2014). For such a network, given a sequence of input words, and the softmax prediction for the next word, we want to identify the importance of the preceding words for the score.
As in the case of the Inception model, we observe saturation in this LSTM network. To describe the setup, we choose 20 randomly chosen sections of the test data, and for each of them inspect the prediction score of the next word using the first 10 words. Then we give each of the 10 input words a weight of α ∈ [0, 1], which is applied to scale their embedding vectors. In Figure 8, we plot the prediction score as a function of α. For all except one curves, the curve starts near zero at α = 0, moves around in the middle, stabilizes, and turns flat around α = 1. For the interesting special case where softmax score is non-zero at α = 0, it turns out that that the word being predicted represents out of vocabulary words. [!h]
In Table 9 and Table 10 we show two comparisons of gradients to integrated gradients. Due to saturation, the magnitudes of gradients are so small compared to the prediction scores that it is difficult to make sense of them. In comparison, (approximate) integrated gradients have a total amount close to the prediction, and seem to make sense. For example, in the first example, the integrated gradients attribute the prediction score of “than“ to the preceding word “more”. This makes sense as “than” often follows right after “more“ in English. On the other hand, standard gradient gives a slightly negative attribution that betrays our intuition. In the second example, in predicting the second “ual”, integrated gradients are clearly the highest for the first occurrence of
“ual”, which is the only word that is highly predictive of the second “ual”. On the other hand, standard gradients are not only tiny, but also similar in magnitude for multiple words.
4 CONCLUSION
We present Interior Gradients, a method for quantifying feature importance. The method can be applied to a variety of deep networks without instrumenting the network, in fact, the amount of code required is fairly tiny. We demonstrate that it is possible to have some understanding of the performance of the network without a detailed understanding of its implementation, opening up the possibility of easy and wide application, and lowering the bar on the effort needed to debug deep networks.
We also wonder if Interior Gradients are useful within training as a measure against saturation, or indeed in other places that gradients are used.
ACKNOWLEDGMENTS
We would like to thank Patrick Riley and Christian Szegedy for their helpful feedback on the technique and on drafts of this paper.
A APPENDIX
B ATTRIBUTION COUNTER-EXAMPLES
We show that the methods DeepLift and Layer-wise relevance propagation (LRP) break the implementation invariance axiom, and the Deconvolution and Guided back-propagation methods break the sensitivity axiom.
Figure 14 provides an example of two equivalent networks f(x1, x2) and g(x1, x2) for which DeepLift and LRP yield different attributions.
First, observe that the networks f and g are of the form f(x1, x2) = ReLU(h(x1, x2)) and f(x1, x2) = ReLU(k(x1, x2)) 7, where
h(x1, x2) = ReLU(x1)− 1− ReLU(x2) k(x1, x2) = ReLU(x1 − 1)− ReLU(x2)
Note that h and k are not equivalent. They have different values whenever x1 < 1. But f and g are equivalent. To prove this, suppose for contradiction that f and g are different for some x1, x2. Then it must be the case that ReLU(x1) − 1 6= ReLU(x1 − 1). This happens only when x1 < 1, which implies that f(x1, x2) = g(x1, x2) = 0.
Now we leverage the above example to show that Deconvolution and Guided back-propagation break sensitivity. Consider the network f(x1, x2) from Figure 14. For a fixed value of x1 greater than 1, the output decreases linearly as x2 increases from 0 to x1 − 1. Yet, for all inputs, Deconvolutional networks and Guided back-propagation results in zero attribution for x2. This happens because for all inputs the back-propagated signal received at the node ReLU(x2) is negative and is therefore not back-propagated through the ReLU operation (per the rules of deconvolution and guided backpropagation; see Springenberg et al. (2014) for details). As a result, the feature x2 receives zero attribution despite the network’s output being sensitive to it.
7 ReLU(x) is defined as max(x, 0).

The authors propose to measure “feature importance”, or specifically, which pixels contribute most to a network’s classification of an image. A simple (albeit not particularly effective) heuristic for measuring feature importance is to measure the gradients of the predicted class wrt each pixel in an input image I. This assigns a score to each pixel in I (that ranks how much the output prediction would change if a given pixel were to change). In this paper, the authors build on this and propose to measure feature importance by computing gradients of the output wrt scaled version of the input image, alpha*I, where alpha is a scalar between 0 and 1, then summing across all values of alpha to obtain their feature importance score. Here the scaling is simply linear scaling of the pixel values (alpha=0 is all black image, alpha=1 is original image). The authors call these scaled images “counterfactuals” which seems like quite an unnecessarily grandiose name for literally, a scaled image.
The authors show a number of visualizations that indicate that the proposed feature importance score is more reasonable than just looking at gradients only with respect to the original image. They also show some quantitative evidence that the pixels highlighted by the proposed measure are more likely to fall on the objects rather than spurious parts of the image (in particular, see figure 5). The method is also applied to other types of networks. The quantitative evidence is quite limited and most of the paper is spent on qualitative results.
While the goal of understanding deep networks is of key importance, it is not clear whether this paper really help elucidate much. The main interesting observation in this paper is that scaling an image by a small alpha (i.e. creating a faint image) places more “importance” on pixels on the object related to the correct class prediction. Beyond that, the paper builds a bit on this, but no deeper insight is gained. The authors propose some hand-wavy explanation of why using small alpha (faint image) may force the network to focus on the object, but the argument is not convincing. It would have been interesting to try to probe a bit deeper here, but that may not be easy.
Ultimately, it is not clear how the proposed scheme for feature importance ranking is useful. First, it is still quite noisy and does not truly help understand what a deep net is doing on a particular image. Performing a single gradient descent step on an image (or on the collection of scaled versions of the image) hardly begins to probe the internal workings of a network. Moreover, as the authors admit, the scheme makes the assumption that each pixel is independent, which is clearly false.
Considering the paper presents a very simple idea, it is far too long. The main paper is 14 pages, up to 19 with references and appendix. In general the writing is long-winded and overly verbose. It detracted substantially from the paper. The authors also define unnecessary terminology. “Gradients of Coutnerfactuals” sounds quite fancy, but is not very related to the ideas explored in the writing. I would encourage the authors to tighten up the writing and figures down to a more readable page length, and to more clearly spell out the ideas explored early on.