Reading comprehension is a question answering task where the answer is to be found in a given passage about entities and events not mentioned in general knowledge sources. A significant number of neural architectures for this task (neural readers) have recently been developed and evaluated on large cloze-style datasets. We present experiments supporting the emergence of “predication structure” in the hidden state vectors of a class of neural readers including the Attentive Reader and Stanford Reader. We posits that the hidden state vectors can be viewed as (a representation of) a concatenation [P, c] of a “predicate vector” P and a “constant symbol vector” c and that the hidden state represents the atomic formula P (c). This predication structure plays a conceptual role in relating “aggregation readers” such as the Attentive Reader and the Stanford Reader to “explicit reference readers” such as the Attention-Sum Reader, the Gated-Attention Reader and the Attention-over-Attention Reader. In an independent contribution, we show that the addition of linguistics features to the input to existing neural readers significantly boosts performance yielding the best results to date on the Who-did-What dataset.1
1 INTRODUCTION AND OVERVIEW
Reading comprehension is a type of question answering task where the answer is to be found in a passage about particular entities and events not otherwise familiar to the reader. In particular, the entities and events should not be mentioned in structured databases of general knowledge. Reading comprehension problems are intended to measure a systems ability to extract semantic information about entities and relations directly from unstructured text. Several large scale reading comprehension datasets have been introduced recently. In particular the CNN & DailyMail datasets (Hermann et al., 2015), the Children’s Book Test (CBT) (Hill et al., 2016), and the Who-did-What dataset (Onishi et al., 2016). The large sizes of these datasets enable the application of deep learning. These are all cloze-style datasets where a question is constructed by deleting a word or phrase from an article summary (in CNN/DailyMail), from a sentence in a Children’s story (in CBT), or by deleting a person from the first sentence of a different news article on the same entities and events (in Who-did-What).
In this paper we present empirical evidence for the emergence of predication structure in a certain class of neural readers. To understand predication structure is it helful to review the anonymization performed in the CNN/DailyMail dataset. In this dataset named entities are replaced by anonymous entity identifiers such as “entity37”. The passage might contain “entity52 gave entity24 a rousing applause” and the question might be “X received a rounding applause from entity52”. The task is to fill in X from a given multiple choice list of candidate entity identifiers. A fixed relatively small set of the same entity identifiers are used over all the problems and the same problem is presented many times with the entity identifiers shuffled. This prevents a given entity identifier from having any semantically meaningful vector embedding. The embeddings of the entity identifiers are
∗Authors contributed equally 1code will be available: https://github.com/sohuren
presumably just pointers to semantics-free tokens. We will write entity identifiers as logical constant symbols such as c rather than strings such as “entity37”.
Aggregation readers, including Memory Networks (Weston et al.; Sukhbaatar et al., 2015), the Attentive Reader (Hermann et al., 2015) and the Stanford Reader (Chen et al., 2016), use bidirectional LSTMs or GRUs to construct a contextual embedding ht of each position t in the passage and also an embedding q of the question. They then select and answer c using a criterion similar to
argmax c ∑ t < ht, q > < ht, e(c) > (1)
where e(c) is the vector embedding of the constant symbol (entity identifier) c. In practice the inner-product < ht, q > is normalized over t using a softmax to yield an attention αt over t and (1) becomes.
argmax c < e(c), ∑ t αtht > . (2)
Here ∑ t αtht is viewed as a vector representation of the passage.
We argue that for aggregation readers, roughly defined by (2), the hidden state ht of the passage at position (or word) t can be viewed as a vector concatenation ht = [e(Φt), e′(ct)] where Φt is a property (or statement or predicate) being stated of a particular constant symbol ct. A logician might write this as ht = Φt[ct]. Furthermore, the question can be interpreted as having the form Ψ[x] where the problem is to find a constant symbol c such that the passage implies Ψ[c]. Assuming ht = [e(Φt), e ′(ct)] and q = [e(Ψ), 0] and e(c) = [0, e′(c)] we can rewrite (1) as
argmax c ∑ t < e(Φt), e(Ψ) > < e ′(ct), e ′(c) > . (3)
The first inner product in (3) is interpreted as measuring the extent to which Φt[x] implies Ψ[x] for any x. The second inner product is interpreted as restricting t to positions talking about the constant symbol c.
Note that the posited decomposition of ht is not explicit in (2) but instead must emerge during training. We present empirical evidence that this structure does emerge. The empirical evidence is somewhat tricky as the direct sum structure that divides ht into its two parts need not be axis aligned and therefore need not literally correspond to vector concatenation.
We also consider a second class of neural readers that we call explicit reference readers. Explicit reference readers avoid (2) and instead use
argmax c ∑ t∈R(c) αt (4)
where R(c) is the subset of the positions where the constant symbol (entity identifier) c occurs. Note that if we identify αt with < e(Φt), e(Ψ) > and assume that < e′(c), e′(ct) > is either 0 or 1 depending on whether c = ct, then (3) and (4) agree. In explicit reference readers the hidden state ht need not carry a pointer to ct as the restriction on t is independent of learned representations. Explicit reference readers include the Attention Sum Reader (Kadlec et al., 2016), the Gated Attention Reader (Dhingra et al., 2016), the Attention-over-Attention Reader (Cui et al., 2016) and others (a list can be found in section 6).
So far we have only considered anonymized datasets that require the handling of semantics-free constant symbols. However, even for non-anonymized datasets such as Who-Did-What, it is helpful to add features which indicate which positions in the passage are referring to which candidate answers. This indicates, not surprisingly, that reference is important in question answering. The fact that explicit reference features are needed in aggregation readers on non-anonymized data indicates that reference is not being solved by the aggregation readers. However, as reference seems to be important for cloze-style question answering, these problems may ultimately provide training data from which reference resolution can be learned.
Sections 2 and 3 review various existing datasets and models respectively. Section 4 presents the logical structure interpretation of aggregation readers in more detail and the empirical evidence supporting it. Section 5 proposes new models that enforce the direct sum structure of the hidden
state vectors. It is shown that these new models perform well on the Who-did-What dataset provided that reference annotations are added as input features. Section 5 also describes additional linguistic features that can be added to the input embeddings and show that these improve the performance of existing models resulting in the best single-model performance to date on the Who-did-What dataset.
2 A BRIEF SURVEY OF DATASETS
Before presenting various models for machine comprehension we give a general formulation of the machine comprehension task. We take an instance of the task be a four tuple (q, p, a,A), where q is a question given as sequence of words containing a special taken for a “blank” to be filled in, p is a document consisting of a sequence of words, A is a set of possible answers and a ∈ A is the ground truth answer. All words are drawn from a vocabulary V . We assume that all possible answers are words from the vocabulary, that is A ⊆ V , and that the ground truth answer appears in the document, that is a ∈ p. The problem can be described as that of selecting the answer a ∈ A that answers question q based on information from p.
We will now briefly summarize important features of the related datasets in reading comprehension.
CNN & DailyMail: Hermann et al. (2015) constructed these datasets from a large number of news articles from the CNN and Daily Mail news websites. The main article is used as the context, while the cloze style question is formed from one short highlight sentence appearing in conjunction with the published article. To avoid the model using external world knowledge when answering the question, the named entities in the entire dataset were replaced by anonymous entity IDs which were then further shuffled for each example. This forces models to rely on the context document to answer each question. In this anonymized corpus the entity identifiers are taken to be a part of the vocabulary and the answer set A consists of the entity identifiers occurring in the passage. Who-did-What (WDW): The Who-did-What dataset (Onishi et al., 2016) contains 127,000 multiple choice cloze questions constructed from the LDC English Gigaword newswire corpus (David & Cieri, 2003). In contrast with CNN and Daily Mail, it avoids using article summaries for question formation. Instead, each problem is formed from two independent articles: one is given as the passage to be read and a different article on the same entities and events is used to form the question. Further, Who-did-What avoids anonymization, as each choice is a person named entity. In this dataset the answer set A consists of the person named entities occurring in the passage. Finally, the problems have been filtered to remove a fraction that are easily solved by simple baselines. It has two training sets. The larger training set (“relaxed”) is created using less baseline filtering, while the smaller training set (“strict”) uses the same filtering as the validation and test sets.
Children’s Book Test (CBT) Hill et al. (2016) developed the CBT dataset in a slightly different fashion to the CNN/DailyMail datasets. They take any sequence of 21 consecutive sentences from a children’s book: the first 20 sentences are used as the passage, and the goal is to infer a missing word in the 21st sentence. The task complexity varies with the type of the omitted word (verb, preposition, named entity, or common noun). According to the original study on this dataset (Hill et al., 2016), n-gram and recurrent neural network language models are sufficient for predicting verbs or prepositions. However, for named entities and common nouns, current solvers are still far from human performance.
Other Related Datasets. It is also worth mentioning several related datasets. The MCTest dataset (Richardson et al., 2013) consists of children’s stories and questions written by crowdsourced workers. The dataset only contains 660 documents and is too small to train deep models. The bAbI dataset (Weston et al., 2016) is constructed automatically using synthetic text generation and can be perfectly answered by hand-written algorithms (Lee et al., 2016). The SQuAD dataset (Rajpurkar et al., 2016) consists passage-question pairs where the passage is a wikipedia article and the questions are written by crowdsourced workers. Although crowdsourcing is involved, the dataset contains over 200,000 problems. But the answer is often a word sequence which is dificult to handle with the reader models considered here. The LAMBADA dataset (Denis et al., 2016) is a word prediction dataset which requires a broad discourse context and the correct answer might not in the context. Nonetheless, when the correct answer is in the context, neural readers can be applied effectively (Chu et al., 2016).
3 AGGREGATION READERS AND EXPLICIT REFERENCE READERS
Here we classify readers into aggregation readers and explicit reference readers. Aggregation readers appeared first in the literature and include Memory Networks (Weston et al.; Sukhbaatar et al., 2015), the Attentive Reader (Hermann et al., 2015), and the Stanford Reader (Chen et al., 2016). Aggregation readers are defined by equations (8) and (10) below. Explicit reference readers incluce the Attention-Sum Reader (Kadlec et al., 2016), the Gated-Attention Reader (Dhingra et al., 2016), and the Attention-over-Attention Reader (Cui et al., 2016). Explicit reference readers are defined by equation (14) below. We first present the Stanford Reader as a paradigmatic aggregation Reader and the Attention-Sum Reader as a paradigmatic explicit reference reader.
3.1 AGGREGATION READERS
Stanford Reader. The the Stanford Reader (Chen et al., 2016) computes a bi-directional LSTM representation of both the passage and the question.
h = biLSTM(e(p)) (5) q = [fLSTM(e(q))|q|, bLSTM(e(q))1] (6)
In equations (5) and (6) we have that e(p) is the sequence of word embeddings e(wi) for wi ∈ p and similarly for e(q). The expression biLSTM(s) denotes the sequence of hidden state vectors resulting from running a bi-directional LSTM on the vector sequence s. We write biLSTM(s)i for the ith vector in this sequence. Similarly fLSTM(s) and bLSTM(s) denote the sequence of vectors resulting from running a forward LSTM and a backward LSTM respectively and [·, ·] denotes vector concatenation. The Stanford Reader, and various other readers, then compute a bilinear attention over the passage which is then used to construct a single weighted vector representation of the passage.
αt = softmax t
h>t Wα q (7) o = ∑ t αtht (8)
Finally, they compute a probability distribution over the answers P (a|p, q,A).
p(a|d, q,A) = softmax a∈A eo(a) >o (9)
â = argmax a∈A
eo(a) >o (10)
Here eo(a) is an “output embedding” of the answer a. On the CNN dataset the Stanford Reader trains an output embedding for each the roughly 500 entity identifiers used in the dataset. In cases where the answer might be any word in V an output embedding must be trained for the entire vocabulary. The reader is trained with log-loss ln 1/P (a|p, q,A) where a is the correct answer. At test time the reader is scored on the percentage of problems where â = a.
Memory Networks. Memory Networks (Weston et al.; Sukhbaatar et al., 2015) use (8) and (10) but have more elaborate methods of constructing “memory vectors” ht not involve LSTMs. Memory networks use (8) and (10) but replace (9) with
P (w|p, q,A) = P (w|p, q) = softmax w∈V eo(w) T o. (11)
It should be noted that (11) trains output vectors over the whole vocabulary rather than just those items occurring in the choice setA. This is empirically significant in non-anonymized datasets such as CBT and Who-did-What where choices at test time may never have occurred as choices in the training data.
Attentive Reader. The Stanford Reader was derived from the Attentive Reader (Hermann et al., 2015). The Attentive Reader uses αt = softmaxt MLP([ht, q]) instead of (7). Here MLP(x) is the output of a multi layer perceptron (MLP) given input x. Also, the answer distribution in the attentive reader is defined over the full vocabulary rather than just the candidate answer set A.
P (w|p, q,A) = P (w|p, q) = softmax w∈V eo(w) TMLP([o, q]) (12)
Equation (12) is similar to (11) in that it leads to the training of output vectors for the full vocabulary rather than just those items appearing in choice sets in the training data. As in memory networks, this leads to improved performance on non-anonymized data sets.
3.2 EXPLICIT REFERENCE READERS
Attention-Sum Reader. In the Attention-Sum Reader (Kadlec et al., 2016) h and q are computed with equations (5) and (6) as in the Stanford Reader but using GRUs rather than LSTMs. The attention αt is computed similarly to (7) but using a simple inner product αt = softmaxt h>t q rather than a trained bilinear form. Most significanlty, however, equations (9) and (10) are replaced by the following where t ∈ R(a, p) indicates that a reference to candidate answer a occurs at position t in p.
P (a|p, q,A) = ∑
t∈R(a,p)
αt (13)
â = argmax a ∑ t∈R(a,p) αt (14)
Here we think of R(a, p) as the set of references to a in the passage p. It is important to note that (13) is an equality and that P (a|p, q,A) is not normalized to the members ofR(a, p). When training with the log-loss objective this drives the attention αt to be normalized — to have support only on the positions t with t ∈ R(a, p) for some a. See the heat maps in the appendix. Gated-Attention Reader. The Gated Attention Reader Dhingra et al. (2016) involves a K-layer biGRU architecture defined by the following equations.
q` = [fGRU(e(q))|q|, bGRU(e(q))1] 1 ≤ ` ≤ K h1 = biGRU(e(p))
h` = biGRU(h`−1 q`−1) 2 ≤ ` ≤ K
Here the question embeddings q` for different values of ` are computed with different GRU model parameters. Here h q abbreviates the sequence h1 q, h2 q, . . . h|p| q. Note that for K = 1 we have only q1 and h1 as in the attention-sum reader. An attention is then computed over the final layer hK with αt = softmaxt (hKt )
> qK in the attention-sum reader. This reader uses (13) and (14).
Attention-over-Attention Reader, The Attention-over-Attention Reader (Cui et al., 2016) uses a more elaborate method to compute the attention αt. We will use t to range over positions in the passage and j to range over positions in the question. The model is then defined by the following equations.
h = biGRU(e(p)) q = biGRU(e(q))
αt,j = softmaxt h > t qj βt,j = softmaxj h > t qj
βj = 1 |p| ∑ t βt,j αt = ∑ j βjαt,j
Note that the final equation defining αt can be interpreted as applying the attention βj to the attentions αt,j . This reader uses (13) and (14).
4 EMERGENT PREDICATION STRUCTURE
As discussed in the introduction the entity identifiers such as “entity37” introduced in the CNN/DailyMail dataset cannot be assigned any semantics other than their identity. We should think of them as pointers or semantics-free constant symbols. Despite this undermining of semantics, aggregation readers using (8) and (10) are able to perform well. Here we posit that this is due to an emergent predication structure in the hidden vectors ht. Intuitively we want to think of the hidden state vector ht as a concatenation [e(Φt), e′o(at)] where Φt carries semantic information true of at. We think of ht as representing Φt[at] for semantic statement Φt[x] asserted of the constant symbol
at. We also think of the vector representation q of the question as having the form [e(Ψ), 0] and the vector embedding eo(a) as having the form [0, e′o(a)].
Unfortunately, the decomposition of ht into this predication structure need not be axis aligned. Rather than posit an axis-aligned concatenation we posit that the hidden vector spaceH is a possibly non-aligned direct sum H = S ⊕ E (15) where S is a subspace of “statement vectors” and E is an orthogonal subspace of “entity pointers”. Each hidden state vector h ∈ H then has a unique decomposition as h = Ψ+e for Ψ ∈ S and e ∈ E. This is equivalent to saying that the hidden vector space H is some rotation of a concatenation of the vector spaces S and E.
We now present empirical evidence for this decomposition structure. We first note that the predication decomposition implies that eo(a)>ht equals eo(a)>eo(at). This suggests the following for some fixed positive constant c.
eo(a) >ht = { c if t ∈ R(a, p) 0 otherwise (16)
Assuming the predication structure we have c = ||eo(a)||2. We note that if different entity constants had different norms then answers would be biased toward occurrences of the constant symbol of larger norm. But we need to have that all constant symbols are equivalent. We note that (??) gives
argmax a
eo(a) >o = argmax
a eo(a) > ∑ t αtht
= argmax a ∑ t αt eo(a) >ht = argmax a ∑ t∈R(a,p) αt
and hence (10) and (14) agree — the aggregation readers and the explicit reference readers are using essentially the same answer selection criterion.
Empirical evidence for (16) is given in the first three rows of table 1. The first row empirically measures the “constant” c in (16) by measuring e0(a)>ht for those cases where t ∈ R(a, p). The second row measures “0” in (16) by measuring eo(a)>ht in those cases where t 6∈ R(a, p). Additional evidence for (16) is given in figure 1 showing that the output vectors eo(a) for different entity identifiers a are nearly orthogonal. Orthogonality of the output vectors is required by (16) provided that each output vector eo(a) is in the span of the hidden state vectors ht,p for which t ∈ R(a, p). Intuitively, the mean of all vectors ht,p with t ∈ R(a, p) should be approximately equal to eo(a). Of course empirically this will only be approximately true.
Equation (16) would suggest that the vector embedding of the constant symbols should have dimension at least as large as the number of distinct constants. However, in practice is sufficient that e(a)>e(a′) is small for a 6= a′. This allows the vector embeddings of the constants to have dimension much smaller than the number of constants. We have experimented with two-sparse constant symbol embeddings where the number of embedding vectors in dimention d is 2d(d− 1) (d choose 2 times the four ways of setting the signs of the non-zero coordinates). Although we do not report results here, these designed and untrained constant embeddings worked reasonably well.
As further support for (16) we give heat maps for eo(a)ht in the appendix for different identifiers a and heat maps for αt for different readers in the appendix.
As another testable predication we note that the posited decomposition of the hidden state vectors implies
q>(hi + eo(a)) = q >hi. (17)
This equation is equivalent to q>eo(a) = 0. Experimentally, however, we cannot expect q>eo(a) to be exactly zero and (17) seems to provides a more experimentally meaningful test. Empirical evidence for (17) is given in the fourth and fifth row of table 1. The fourth row measures the cosine of the angle between the question vector q and the hidden state ht averaged over passage positions t at which some entity identifier occurs. The fifth row measures the cosine of the angle between q and eo(a) averaged over the entity identifiers a.
A question asks for a value of x such that a statement Ψ[x] is implied by the passage. For a question Ψ we might even suggest the following vectorial interpretation of entailment.
Φ[x] implies Ψ[x] iff Φ>Ψ ≥ ||Ψ||1. This interpretation is exactly correct if some of the dimensions of the vector space correspond to predicates, Ψ is a 0-1 vector representing a conjunction predicates, and Φ is also 0-1 on these dimensions indicating whether a predicate is implied by the context. Of course in practice one expects the dimension to be smaller than the number of possible predicates.
5 POINTER ANNOTATION READERS
It is of course important to note that anonymization provides reference information — anonymization assumes that one can determine coreference so as to replace coreferent phrases with the same entity identifier. Anonymization allows the reference set R(a, p) to be directly read off of the passage. Still, an aggregation reader must learn to recover this explicit reference structure.
Aggregation readers can have difficulty when anonymization is not done. The Stanford Reader achieves just better than 45% on Who-did-What dataset while Attention Sum Reader can get near 60%. But if we anonymize the Who-did-What dataset and then re-train the Stanford Reader, the accuracy jumps to near 65%. Anonymization has two effects. First, it greatly reduces the number of output word eo(a) to be learned — we need only learn output embeddings for the relatively small number of entity identifiers needed. Second, anonymization suppresses the semantics of the reference phrases and leaves only a semantics-free entity identifier. This suppression of semantics may facilitate the separation of the hidden state vector space H into a direct sum S ⊕E with q ∈ S and eo(a) ∈ E. We can think of anonymization as providing additional linguistic input for the reader — it explicitly marks positions of candidate answers and establishes coreference. A natural question is whether
this information can be provided without anonymization by simply adding additional coreference features to the input. Here we evaluate two architectures inspired by this question. This evaluation is done on the Who-did-What dataset which is not anonymized. In each architecture we add features to the input to mark the occurrences of candidate answers. These models are simpler than the Stanford reader but perform comparably. This comparable performance in table 2 further supports our analysis of logical structure in aggregation readers.
One-Hot Pointer Annotation: The Stanford Reader involves both input embeddings of words and output embeddings of entity identifiers. In the Who-did-What dataset each problem has at most five choices in the multiple choice answer list. This means that we need only five entity identifiers and we can use a five dimensional one-hot vector representation for answer identifiers. If an answer choice exists at position t in the passage let it be the index of that choice on the choice list. If no choice occurs t take it to be zero. Take e′(i) to be the zero vector if i = 0 and otherwise to be the one-hot vector for i. We defined pointer annotation to be the result of adding e′(it) as additional features to the input embedding.
e(wt) = [e(wt), e ′(it)] (18)
We then define a one-hot pointer reader by designates five dimensions of the hidden state as indicators of the answer and take the probability of choice i to be defined as
p(i|d, q) = softmax i oi (19)
where o is computed by (8).
General Pointer Annotation: In the CNN dataset there are roughly 500 entity identifier and a onehot representation is not desirable. Instead we can let e′(i) be a fixed set of “pointers vectors” — vectors distributed widely on the unit sphere so that for i 6= j we have that e′(i)>e′(j) is small. We again use (18) but replace (19) with
p(i|d, q) = softmax i
[0, e′(i)]>o (20)
In the general pointer reader the pointer embeddings e′(i) are held fixed and not trained.
Linguistic Features. Each model can be modified to include additional input features for each input token in the question and passage. More specifically we can add the following features to the word embeddings.
• Binary feature: whether current token occurs in the question. • Real value feature: the frequency of current token in the passage.
• Real value feature: position of the token’s first occurrence in the passage as a percentage of the passage length.
• Binary feature: whether the text surrounding token match the text surrounding the placeholder in the question. We only have features for matching both left and right one word.
• One hot vector: Part-of-speech (POS) tagging. We only use such feature on CBT dataset.
• One hot vector: Name Entity Recognition (NER). We only use such feature on CBT dataset.
6 A SURVEY OF RECENT RESULTS
The performance of various recent readers on CNN, DailyMail and CBTest are summarized in Table 3. For purposes of comparison we only present results on single models. Model ensembles generally perform better than single models but are require more computation to train making comparisons more difficult. More experimental details can be found in appendix.
In table 3, all the high-performance approaches are proposed very recently. Blue color represents the second highest accuracy and bold font indicates the state-of-the-art accuracy. Note that the result of Stanford Reader we report here is the one without relabeling since relabeling procedure doesn’t follow the protocol used in Hermann et al. (2015).
7 DISCUSSION
Explicit reference architectures rely on reference resolution — a specification of which phrases in the given passage refer to candidate answers. Our experiments indicate that all existing readers benefit greatly from this externally provided information. Aggregation readers seem to demonstrate a stronger learning ability in that they essentially learn to mimic explicit reference readers by identifying reference annotation and using it appropriately. This is done most clearly in the pointer reader architectures. Furthermore, we have argued for, and given experimental evidence for, an interpretation of aggregation readers as learning emergent logical structure — a factoring of neural representations into a direct sum of a statement (predicate) representation and an entity (argument) representation.
At a very high level our analysis and experiments support a central role for reference resolution in reading comprehension. Automating reference resolution in neural models, and demonstrating its value on appropriate datasets, would seem to be an important area for future research.
Of course there is great interest in “learning representations”. The current state of the art in reading comprehension is such that systems still benefit from externally provided linguistic features including externally annotated reference resolution. It would seem desirable to develop fully automated neural readers that perform as well as readers using externally provided annotations. It is of course important to avoid straw man baselines when making any such claim.
We are hesitant to make any more detailed comments on the differences between the architectural details of the readers discussed in this paper. The differences in scores between the leading readers are comparable to differences in scores that can be achieved by aggressive search over meta parameters or the statistical fluctuations in the quality of models learned by noisy statistical training procedures. More careful experiments over a longer period of time are needed. More dramatic improvements in performance would of course provide better support for particular innovations.
ACKNOWLEDGMENTS
We thanks the support of NVIDIA Corporation with the donation of GPUs used for this work.
8 APPENDIX
8.1 EXPERIMENT DETAILS
We implemented the neural readers using Theano (de ric Bastien et al., 2012) and Blocks (van Merrienboer et al., 2015) and train them on single Nvidia Tesla K40 GPU. Negative log-likelihood is employed as training criterion. We used stochastic gradient descent (SGD) with the ADAM update rule (Kingma & Ba, 2015) and set the learning rate 0.0005.
For Stanford Reader and One-Hot Pointer Reader, we simply follows the Stanford Reader’s setting and didn’t tune it on each dataset. For gated attention reader, the lookup table was randomly initialized with uniform distribution from the interval [-0.2, 0.2] on CBT dataset, but on CNN&DailyMail, the lookup table was initialized by Glove vector (Jeffrey et al., 2014) trained on the train&validatation set (we found that the pre-trained word vector doesn’t improve the accuracy but will accelerate the training) on CNN&DailyMail. On WDW dataset, the lookup table was initialized by pre-trained Glove vector2. It should be noticed that if we initialize the lookup table with pre-trained Glove vector from //nlp.stanford.edu/data/glove.6B.zip, it will slightly boost the accuracy compared with using the Glove vector trained on train&validation set. Input to hidden state weights were initialized by random orthogonal matrices (Saxe et al., 2013) and biases were initialized to zero. Hidden to hidden state weights were initialized by identity matrices to force the model can remember longer information. To compute the attention weight, we αt = ht>Wαq and initialize Wα with random uniform distribution. We also used the gradient clipping (Razvan et al., 2013) with threshold of 10 and batches of size 32.
During training we randomly shuffled all examples within each epoch. To speedup training, we always pre-fetched 10 batches worth of examples and sorted them according to document length as did by Kadlec et al. (2016). When trained on CNN, DailyMail and WDW (anonymization case) dataset, we randomly reshuffled the entity identifier to match the procedure proposed in Hermann et al. (2015).
During training we evaluated the accuracy after each epoch and stopped the training when the accuracy on the validation set started decreasing. We tried limiting the vocabulary to the most frequent tokens but didn’t observed any performance improvement compared with using all the distinct tokens as vocabulary. Since part of our experiments need to check the word embedding assignment issues, finally we use all the distinct tokens as vocabulary. To find the optimal embedding and hidden state dimension, we tried several groups of different combinations, the optimal value and corresponding training statistics in Gated Attention readers are summarized in Table. 4. When anonymize the Whodid-What dataset, we can either use simple string match to replace answer in question and story with entity identifier, or we can use Name Entity Recognition(NER) tools3 to detect name entities and then replace the answer name entities in question and story with entity identifier, we found the later one generally will bring 2 % improvement compared with simple string match. More experimental details can be found in code.
8.2 HEAT MAP OF STANFORD READER FOR DIFFERENT ANSWER CANDIDATES
We randomly choose one article from CNN dataset and show softmax(eo(a)ht) for t ∈ [0, |p|] for each answer candidate a in figure.2, figure.3, figure.4, figure.5 and figure.6. Red color indicates
2http://nlp.stanford.edu/data/glove.6B.zip 3http://nlp.stanford.edu/software/CRF-NER.shtml
larger probability and orange indicates smaller probability and the remaining indicates very low probability that can be ignored. From those figures, we can see that our assumption that eo(a) is used to pick up its occurrence is reasonable.
@entity0 ( @entity1 ) six survivors of the @entity0 kosher supermarket siege in january are suing a @entity5 media outlet for what they call dangerous live broadcasting during the hostage - taking . according to @entity0 prosecutor 's spokeswoman @entity10 , the lawsuit was filed march 27 and a preliminary investigation was opened by the prosecutor 's office wednesday . the media outlet , @entity1 affiliate @entity16 , is accused of endangering the lives of the hostages , who were hiding in a cold room during the attack , by broadcasting their location live during the siege . @entity23 in a statement friday said one of its journalists " mentioned only once the presence of a woman hidden inside the @entity27 , on the basis of police sources on the ground . " " immediately , the chief editor felt that this information should not be released . it therefore has subsequently never been repeated on air or posted on - screen . @entity16 regrets that the mention of this information could cause concern to the hostages , as well as their relatives , that their lives were in danger , " the statement said . gunman @entity47 , also suspected in the slaying of a police officer , stormed the @entity27 @entity51 supermarket on january 9 , killing four people and taking others hostage . he was killed in the police operation to end the siege . a 24 - year - old supermarket employee , @entity57 - born @entity56 , was hailed as a hero afterward when it emerged that he had risked his life to hide 15 customers from @entity47 in the cold room . the hostage - taking was the culmination of three days of terror in @entity0 that began with the january 7 shooting of 12 people at the offices of @entity5 satirical magazine @entity69 . the two brothers blamed for that attack , @entity72 and @entity73 , were killed on january 9 after a violent standoff at an industrial site . the terror attacks claimed the lives of 17 people and put @entity5 on a heightened state of alert . @entity1 's @entity80 reported from @entity0 , and @entity81 wrote from @entity82 . @entity1 's @entity83 contributed to this report . query: they hid in a cold room during the attack in @entity0 by gunman @placeholder
Figure 2: Heat map of softmax(eo(a)ht) when a = entity0.
8.3 HEAT MAP OF DIFFERENT READERS
We randomly choose one article from CNN dataset and show the attention map αt = softmax(q>Waht) for different readers (in Attention Sum and Gated Attention Reader, Wα is identity matrix). From figure 7, figure 8 and figure 9, we can see that different readers essential put the weights on the entity identifiers.
@entity0 ( @entity1 ) six survivors of the @entity0 kosher supermarket siege in january are suing a @entity5 media outlet for what they call dangerous live broadcasting during the hostage - taking . according to @entity0 prosecutor 's spokeswoman @entity10 , the lawsuit was filed march 27 and a preliminary investigation was opened by the prosecutor 's office wednesday . the media outlet , @entity1 affiliate @entity16 , is accused of endangering the lives of the hostages , who were hiding in a cold room during the attack , by broadcasting their location live during the siege . @entity23 in a statement friday said one of its journalists " mentioned only once the presence of a woman hidden inside the @entity27 , on the basis of police sources on the ground . " " immediately , the chief editor felt that this information should not be released . it therefore has subsequently never been repeated on air or posted on - screen . @entity16 regrets that the mention of this information could cause concern to the hostages , as well as their relatives , that their lives were in danger , " the statement said . gunman @entity47 , also suspected in the slaying of a police officer , stormed the @entity27 @entity51 supermarket on january 9 , killing four people and taking others hostage . he was killed in the police operation to end the siege . a 24 - year - old supermarket employee , @entity57 - born @entity56 , was hailed as a hero afterward when it emerged that he had risked his life to hide 15 customers from @entity47 in the cold room . the hostage - taking was the culmination of three days of terror in @entity0 that began with the january 7 shooting of 12 people at the offices of @entity5 satirical magazine @entity69 . the two brothers blamed for that attack , @entity72 and @entity73 , were killed on january 9 after a violent standoff at an industrial site . the terror attacks claimed the lives of 17 people and put @entity5 on a heightened state of alert . @entity1 's @entity80 reported from @entity0 , and @entity81 wrote from @entity82 . @entity1 's @entity83 contributed to this report . query: they hid in a cold room during the attack in @entity0 by gunman @placeholder
Figure 3: Heat map of softmax(eo(a)ht) when a = entity1.
@entity0 ( @entity1 ) six survivors of the @entity0 kosher supermarket siege in january are suing a @entity5 media outlet for what they call dangerous live broadcasting during the hostage - taking . according to @entity0 prosecutor 's spokeswoman @entity10 , the lawsuit was filed march 27 and a preliminary investigation was opened by the prosecutor 's office wednesday . the media outlet , @entity1 affiliate @entity16 , is accused of endangering the lives of the hostages , who were hiding in a cold room during the attack , by broadcasting their location live during the siege . @entity23 in a statement friday said one of its journalists " mentioned only once the presence of a woman hidden inside the @entity27 , on the basis of police sources on the ground . " " immediately , the chief editor felt that this information should not be released . it therefore has subsequently never been repeated on air or posted on - screen . @entity16 regrets that the mention of this information could cause concern to the hostages , as well as their relatives , that their lives were in danger , " the statement said . gunman @entity47 , also suspected in the slaying of a police officer , stormed the @entity27 @entity51 supermarket on january 9 , killing four people and taking others hostage . he was killed in the police operation to end the siege . a 24 - year - old supermarket employee , @entity57 - born @entity56 , was hailed as a hero afterward when it emerged that he had risked his life to hide 15 customers from @entity47 in the cold room . the hostage - taking was the culmination of three days of terror in @entity0 that began with the january 7 shooting of 12 people at the offices of @entity5 satirical magazine @entity69 . the two brothers blamed for that attack , @entity72 and @entity73 , were killed on january 9 after a violent standoff at an industrial site . the terror attacks claimed the lives of 17 people and put @entity5 on a heightened state of alert . @entity1 's @entity80 reported from @entity0 , and @entity81 wrote from @entity82 . @entity1 's @entity83 contributed to this report . query: they hid in a cold room during the attack in @entity0 by gunman @placeholder
Figure 4: Heat map of softmax(eo(a)ht) when a = entity16.
@entity0 ( @entity1 ) six survivors of the @entity0 kosher supermarket siege in january are suing a @entity5 media outlet for what they call dangerous live broadcasting during the hostage - taking . according to @entity0 prosecutor 's spokeswoman @entity10 , the lawsuit was filed march 27 and a preliminary investigation was opened by the prosecutor 's office wednesday . the media outlet , @entity1 affiliate @entity16 , is accused of endangering the lives of the hostages , who were hiding in a cold room during the attack , by broadcasting their location live during the siege . @entity23 in a statement friday said one of its journalists " mentioned only once the presence of a woman hidden inside the @entity27 , on the basis of police sources on the ground . " " immediately , the chief editor felt that this information should not be released . it therefore has subsequently never been repeated on air or posted on - screen . @entity16 regrets that the mention of this information could cause concern to the hostages , as well as their relatives , that their lives were in danger , " the statement said . gunman @entity47 , also suspected in the slaying of a police officer , stormed the @entity27 @entity51 supermarket on january 9 , killing four people and taking others hostage . he was killed in the police operation to end the siege . a 24 - year - old supermarket employee , @entity57 - born @entity56 , was hailed as a hero afterward when it emerged that he had risked his life to hide 15 customers from @entity47 in the cold room . the hostage - taking was the culmination of three days of terror in @entity0 that began with the january 7 shooting of 12 people at the offices of @entity5 satirical magazine @entity69 . the two brothers blamed for that attack , @entity72 and @entity73 , were killed on january 9 after a violent standoff at an industrial site . the terror attacks claimed the lives of 17 people and put @entity5 on a heightened state of alert . @entity1 's @entity80 reported from @entity0 , and @entity81 wrote from @entity82 . @entity1 's @entity83 contributed to this report . query: they hid in a cold room during the attack in @entity0 by gunman @placeholder
Figure 5: Heat map of softmax(eo(a)ht) when a = entity27.
@entity0 ( @entity1 ) six survivors of the @entity0 kosher supermarket siege in january are suing a @entity5 media outlet for what they call dangerous live broadcasting during the hostage - taking . according to @entity0 prosecutor 's spokeswoman @entity10 , the lawsuit was filed march 27 and a preliminary investigation was opened by the prosecutor 's office wednesday . the media outlet , @entity1 affiliate @entity16 , is accused of endangering the lives of the hostages , who were hiding in a cold room during the attack , by broadcasting their location live during the siege . @entity23 in a statement friday said one of its journalists " mentioned only once the presence of a woman hidden inside the @entity27 , on the basis of police sources on the ground . " " immediately , the chief editor felt that this information should not be released . it therefore has subsequently never been repeated on air or posted on - screen . @entity16 regrets that the mention of this information could cause concern to the hostages , as well as their relatives , that their lives were in danger , " the statement said . gunman @entity47 , also suspected in the slaying of a police officer , stormed the @entity27 @entity51 supermarket on january 9 , killing four people and taking others hostage . he was killed in the police operation to end the siege . a 24 - year - old supermarket employee , @entity57 - born @entity56 , was hailed as a hero afterward when it emerged that he had risked his life to hide 15 customers from @entity47 in the cold room . the hostage - taking was the culmination of three days of terror in @entity0 that began with the january 7 shooting of 12 people at the offices of @entity5 satirical magazine @entity69 . the two brothers blamed for that attack , @entity72 and @entity73 , were killed on january 9 after a violent standoff at an industrial site . the terror attacks claimed the lives of 17 people and put @entity5 on a heightened state of alert . @entity1 's @entity80 reported from @entity0 , and @entity81 wrote from @entity82 . @entity1 's @entity83 contributed to this report . query: they hid in a cold room during the attack in @entity0 by gunman @placeholder
Figure 6: Heat map of softmax(eo(a)ht) when a = entity47.
( @entity3 ) suspected @entity2 militants this week attacked civilians inside @entity5 for the first time in a month , killing at least 16 villagers , a military spokesman told @entity3 saturday . six attackers were killed by @entity5 forces , said maj. @entity10 , an operations officer with a special military unit set up to fight @entity2 . the attackers came thursday " in the hundreds ... torched @entity14 village in the @entity15 , " he said . @entity14 is a village that borders @entity17 and has been identified as a recruiting ground for @entity2 . regional gov. @entity19 said the insurgents have been attacking border villages in @entity5 in search of supplies . @entity5 troops retook cattle that was stolen by the attackers in @entity14 , @entity10 said . the last attack in @entity5 by the @entity29 - based militants was march 10 , when the assailants struck the locality of @entity32 in a failed attempt to overrun a military base . @entity2 , whose name translates as " @entity44 education is sin , " has been waging a years - long campaign of terror aimed at instituting its extreme version of @entity42 law in @entity29 . @entity2 's tactics have intensified in recent years , from battling @entity29 government soldiers to acts disproportionately affecting civilians -- such as raids on villages , mass kidnappings , assassinations , market bombings and attacks on churches and unaffiliated mosques . much of this violence has taken place in @entity29 , but neighboring countries -- @entity5 included -- have also been hit increasingly hard . journalist @entity61 in @entity63 , @entity5 , contributed to this report . query: @placeholder is based in @entity29 but has attacked across the border of several neighbors
Figure 7: Heat map αt for Stanford Reader
( @entity3 ) suspected @entity2 militants this week attacked civilians inside @entity5 for the first time in a month , killing at least 16 villagers , a military spokesman told @entity3 saturday . six attackers were killed by @entity5 forces , said maj. @entity10 , an operations officer with a special military unit set up to fight @entity2 . the attackers came thursday " in the hundreds ... torched @entity14 village in the @entity15 , " he said . @entity14 is a village that borders @entity17 and has been identified as a recruiting ground for @entity2 . regional gov. @entity19 said the insurgents have been attacking border villages in @entity5 in search of supplies . @entity5 troops retook cattle that was stolen by the attackers in @entity14 , @entity10 said . the last attack in @entity5 by the @entity29 - based militants was march 10 , when the assailants struck the locality of @entity32 in a failed attempt to overrun a military base . @entity2 , whose name translates as " @entity44 education is sin , " has been waging a years - long campaign of terror aimed at instituting its extreme version of @entity42 law in @entity29 . @entity2 's tactics have intensified in recent years , from battling @entity29 government soldiers to acts disproportionately affecting civilians -- such as raids on villages , mass kidnappings , assassinations , market bombings and attacks on churches and unaffiliated mosques . much of this violence has taken place in @entity29 , but neighboring countries -- @entity5 included -- have also been hit increasingly hard . journalist @entity61 in @entity63 , @entity5 , contributed to this report . query: @placeholder is based in @entity29 but has attacked across the border of several neighbors
Figure 8: Heat map αt for Gated Attention Reader
( @entity3 ) suspected @entity2 militants this week attacked civilians inside @entity5 for the first time in a month , killing at least 16 villagers , a military spokesman told @entity3 saturday . six attackers were killed by @entity5 forces , said maj. @entity10 , an operations officer with a special military unit set up to fight @entity2 . the attackers came thursday " in the hundreds ... torched @entity14 village in the @entity15 , " he said . @entity14 is a village that borders @entity17 and has been identified as a recruiting ground for @entity2 . regional gov. @entity19 said the insurgents have been attacking border villages in @entity5 in search of supplies . @entity5 troops retook cattle that was stolen by the attackers in @entity14 , @entity10 said . the last attack in @entity5 by the @entity29 - based militants was march 10 , when the assailants struck the locality of @entity32 in a failed attempt to overrun a military base . @entity2 , whose name translates as " @entity44 education is sin , " has been waging a years - long campaign of terror aimed at instituting its extreme version of @entity42 law in @entity29 . @entity2 's tactics have intensified in recent years , from battling @entity29 government soldiers to acts disproportionately affecting civilians -- such as raids on villages , mass kidnappings , assassinations , market bombings and attacks on churches and unaffiliated mosques . much of this violence has taken place in @entity29 , but neighboring countries -- @entity5 included -- have also been hit increasingly hard . journalist @entity61 in @entity63 , @entity5 , contributed to this report . query: @placeholder is based in @entity29 but has attacked across the border of several neighbors
Figure 9: Heat map αt for Attention Sum Reader

The paper aims to consolidate some recent literature in simple types of "reading comprehension" tasks involving matching questions to answers to be found in a passage, and then to explore the types of structure learned by these models and propose modifications. These reading comprehension datasets such as CNN/Daily Mail are on the simpler side because they do not generally involve chains of reasoning over multiple pieces of supporting evidence as can be found in datasets like MCTest. Many models have been proposed for this task, and the paper breaks down these models into "aggregation readers" and "explicit reference readers." The authors show that the aggregation readers organize their hidden states into a predicate structure which allows them to mimic the explicit reference readers. The authors then experiment with adding linguistic features, including reference features, to the existing models to improve performance.
I appreciate the re-naming and re-writing of the paper to make it more clear that the aggregation readers are specifically learning a predicate structure, as well as the inclusion of results about dimensionality of the symbol space. Further, I think the effort to organize and categorize several different reading comprehension models into broader classes is useful, as the field has been producing many such models and the landscape is unclear.
The concerns with this paper are that the predicate structure demonstrated is fairly simple, and it is not clear that it provides insight towards the development of better models in the future, since the "explicit reference readers" need not learn it, and the CNN/Daily Mail dataset has very little headroom left as demonstrated by Chen et al. 2016. The desire for "dramatic improvements in performance" mentioned in the discussion section probably cannot be achieved on these datasets. More complex datasets would probably involve multi-hop inference which this paper does not discuss. Further, the message of the paper is a bit scattered and hard to parse, and could benefit from a bit more focus.
I think that with the explosion of various competing neural network models for NLP tasks, contributions like this one which attempt to organize and analyze the landscape are valuable, but that this paper might be better suited for an NLP conference or journal such as TACL.


DIET NETWORKS: THIN PARAMETERS FOR FAT GENOMICS
Learning tasks such as those involving genomic data often poses a serious challenge: the number of input features can be orders of magnitude larger than the number of training examples, making it difficult to avoid overfitting, even when using the known regularization techniques. We focus here on tasks in which the input is a description of the genetic variation specific to a patient, the single nucleotide polymorphisms (SNPs), yielding millions of ternary inputs. Improving the ability of deep learning to handle such datasets could have an important impact in medical research, more specifically in precision medicine, where highdimensional data regarding a particular patient is used to make predictions of interest. Even though the amount of data for such tasks is increasing, this mismatch between the number of examples and the number of inputs remains a concern. Naive implementations of classifier neural networks involve a huge number of free parameters in their first layer (number of input features times number of hidden units): each input feature is associated with as many parameters as there are hidden units. We propose a novel neural network parametrization which considerably reduces the number of free parameters. It is based on the idea that we can first learn or provide a distributed representation for each input feature (e.g. for each position in the genome where variations are observed in data), and then learn (with another neural network called the parameter prediction network) how to map a feature’s distributed representation (based on the feature’s identity not its value) to the vector of parameters specific to that feature in the classifier neural network (the weights which link the value of the feature to each of the hidden units). This approach views the problem of producing the parameters associated with each feature as a multi-task learning problem. We show experimentally on a population stratification task of interest to medical studies that the proposed approach can significantly reduce both the number of parameters and the error rate of the classifier.
1 INTRODUCTION
Medical datasets often involve a dire imbalance between the number of training examples and the number of input features, especially when genomic information is used as input to the trained pre-
∗Equal contribution.
dictor. This is problematic in the context where we want to apply deep learning (which typically involves large models) to precision medicine, i.e., making patient-specific predictions using a potentially large set of input features to better characterize the patient. This paper proposes a novel approach, called Diet Networks, to reparametrize neural networks to considerably reduce their number of free parameters when the input is very high-dimensional and orders of magnitude larger than the number of training examples.
Genomics is the study of the genetic code encapsulated as DNA in all living organisms’ cells. Genomes contain the instructions to produce and regulate all the functional components needed to guide the development and adaptation of living organisms. In the last decades, advances in genomic technologies resulted in an explosion of available data, making it more interesting to apply advanced machine learning techniques such as deep learning. Learning tasks involving genomic data and already tackled by deep learning include: using Convolutional Neural Networks (CNNs) to learn the functional activity of DNA sequences (Basset package, Kelley et al. (2016), predicting effects of noncoding DNA (DeepSEA, Zhou & Troyanskaya (2015)), investigating the regulatory role of RNA binding proteins in alternative splicing (Alipanahi et al., 2015), inferring gene expression patterns (Chen et al., 2016; Singh et al., 2016) and population genetic parameters (Sheehan & Song, 2016) among others (see Leung et al. (2016) for a detailed example). Noticeably, most of these techniques are based on sequence data where convolutional or recurrent networks are appropriate. When the full DNA sequence is unavailable, such as when data is acquired through genotyping, other methods need to be used. All this work shows that deep learning can be used to tackle genomic-related tasks, paving the road towards a better understanding of the biological impact of DNA variation.
Applying deep learning to human genetic variation holds the promise of identifying individuals at risk for medical conditions. Modern genotyping technologies usually target millions of simple variants across the genome, called single nucleotide polymorphisms (SNPs). These genetic mutations result from substitutions from one nucleotide to another (eg. A to C), where both versions exist within a population. In modern studies, as many as 5 millions SNPs can be acquired for every participant. These datasets differ from other types of genomic data because they focus on the genetic differences between individuals which represents a space of high dimensionality where sequencecontext information is unavailable. In medical genetics, these variants are tested for their association with a trait of interest, an approach termed genome-wide association study (GWAS). This methodology aims at finding genetic variants implicated in disease susceptibility, etiology and treatment.
An important confounding factor in GWAS is population stratification, which arises because both disease prevalence and genetic profiles vary from one population to the other. Although most GWAS have been restricted to homogeneous populations, dimensionality reduction techniques are generally used to account for population-level genetic differences (Price et al., 2006). Our experiments compare such dimensionality reduction techniques (based on principal components analysis, PCA) to the proposed Diet Network parametrization, as well as with standard deep networks.
Recently, several machine learning methods have been successfully applied to detect population stratification, based on the presence of systematic differences in genetic variation between populations. For instance, Support Vector Machines (SVM) models have been used multiple times to infer recent genetic ancestry of sub-continental populations (Haasl et al. (2013)), and local ancestry in admixed populations (SupportMix, Omberg et al. (2012), 23andMe, Inc.). However SVM methods are very sensitive to the the kernel choice and the parameters. They also tend to overfit the model selection criterion which usually induces a limitation in its predictive power.
In this work, we are interested in predicting the genetic ancestry of an individual from their SNP data using a novel deep learning approach, Diet Networks, which allow us to considerably reduce the number of free parameters. Therefore, we propose to tackle this problem by introducing a multi-task architecture in which the problem of predicting the appropriate parameters for each input feature is considered like a task in itself, and the same parameter prediction network is used for all of the hundreds of thousands of input features. This parameter prediction network learns to predict these feature-specific parameters as a function of a distributed representation of the feature identity, or feature embedding. The feature embedding can be learned as part of end-to-end training or using other datasets or a priori knowledge about the features. What is important is that two features which are similar in some appropriate sense (in terms of their interactions with other features or other variables observed in any dataset) end up having similar embeddings, and thus a similar parameter vector as output of the parameter prediction network. A practical advantage of this approach is
that the parameter prediction network can generalize to new features for which there is no labeled training data (without the target to be predicted by the classifier), so long as it is possible to derive an embedding for that feature (for example using just the unlabeled observations of co-occurences of that feature with other features in human genomes).
An interesting consideration is that from the point of the parameter prediction network, each feature is an example: more features now allow to better train the parameter prediction network. It is like if we were considering not the data matrix itself but its transpose. This is actually how the Diet Network implementation processes the data, by using the transpose of the matrix of input values as the input part of the learning task for the parameter prediction network.
The idea of having two networks interacting with each other and with one producing parameters for the other is well rooted in the machine learning literature (Bengio et al., 1991; Schmidhuber, 1992; Gomez & Schmidhuber, 2005; Stanley et al., 2009; Denil et al., 2013; Andrychowicz et al., 2016). Recent efforts in the same direction include works such as (Bertinetto et al., 2016; Brabandere et al., 2016; Ha et al., 2016) that use a network to predict the parameters of a Convolutional Neural Network (CNN). Brabandere et al. (2016) introduce a dynamic filter module that generates network filters conditioned on an input. Bertinetto et al. (2016) propose to learn the parameters of a deep model in one shot, by training a second network to predict the parameters of the first from a single exemplar. Hypernetworks (Ha et al., 2016) explore the idea of using a small network to predict the parameters of another network, training them in an end-to-end fashion. The small network takes as input the feature embedding from the previous layer and learns the parameters of the current layer.
To the best of our knowledge, deep learning has never been used so far to tackle the problem of ancestry prediction based on SNP data. Compared to other approaches that attempt to learn model parameters using a parameter prediction network, our main goal is to reduce the large number of parameters required by the model, by considering the input features themselves as sub-tasks in a multi-task view of the learning problem, as opposed to constructing a model with even higher capacity, as seen, e.g. in (Ha et al., 2016). Our approach is thus based on building an embedding of these tasks (the features) in order to further reduce the number of parameters.
We evaluate our method on a publicly available dataset for ancestry prediction, the 1000 Genomes dataset1, that best represents the human population diversity. Because population-specific differences in disease and drug response are widespread, identifying an individual’s ancestry heritage based on SNP data is a very important task to help detect biological causation and achieve good predictive performance in precision medicine. Most importantly, ancestry-aware approaches in precision genomics will reduce the hidden risks of genetic testing, by preventing spurious diagnosis and ineffective treatment.
2 METHOD
In this section, we describe the Diet Networks as well as the feature embeddings used by the model.
2.1 MODEL
Our model aims at reducing the number of free parameters that a network trained on fat data would typically have.
Let X ∈ RN×Nd be a matrix of data, with N samples and Nd features, where N Nd (e. g. N being approximately 100 times smaller than Nd). We build a multi-layer perceptron (MLP), which takes X as input, computes a hidden representation and outputs a prediction Ŷ. Optionally, the MLP may generate a reconstruction X̂ of the input data from the hidden representation. Figure 1(a) illustrates this basic network architecture. Let xi be one data sample, i.e. a row in X. The standard formulations to compute its hidden representation hi, output prediction ŷi and reconstruction x̂i are given by
hi = f(xi), ŷi = g(hi), x̂i = r(hi), (1)
where f , g and r are non-linear functions.
1http://www.internationalgenome.org/
The number of parameters of the first hidden layer of the architecture grows linearly with the dimensionality of the input data:
h (1) i = f1(xiWe + be), (2)
where We and be are the layer’s parameters. Using fat data such as the one described in Section 1, leads to a parameter explosion in this layer, hereafter referred to as fat hidden layer. To give the reader an intuition, consider the case of having an input with Nd = 300K, and a hidden layer with N1h = 100, the number of parameters of such a layer would be 30M. The same happens to the number of parameters of the optional reconstruction layer, hereafter referred to as fat reconstruction layer.
In order to mitigate this effect, we introduce an auxiliary network to predict the fat layers’ parameters. The auxiliary network takes as input the transposed data matrix XT, extracts a feature embedding and learns a function of this embedding, to be used as parameters of a fat layer:
(We)j: = φ(ej), (3)
where ej represents the embedding of a feature in XT, φ is a non-linear function and (We)j: is the j-th row of We. This means that each feature is associated with the vector of values it takes in the dataset (e.g. across the patients). Other representations could be used, e.g., derived from other datasets in which those features interact. Figure 1(b) shows a prediction network which is an auxiliary network that predicts the parameters of the fat hidden layer of our basic network. Following the same spirit, Figure 1(c) highlights the interaction between a second prediction network that predicts the fat reconstruction layer parameters and the basic network. The architectures of both auxiliary networks may share the initial feature embedding.
The feature embeddings used in the auxiliary networks allow us to substantially reduce the number of free parameters of the fat layers of the basic architecture. The auxiliary network should predict a matrix of weights of size Nd × N1h from a feature embedding. Consider a feature embedding that would transform each N -dimensional feature into a Nf -dimensional vector, where Nf < N . The auxiliary network would learn a function φ : RNf → RN1h . Thus, the fat hidden layer of our basic architecture would have Nf × N1h free parameters (assuming a single layer MLP in the auxiliary network), instead ofNd×N1h . Following our previous example, whereNd = 300K andN1h = 100, using an auxiliary network with previously-obtained feature embeddings of dimensionality Nf = 500 would reduce the number of free parameters of the basic network by a factor of 600 (from 30M to 50K).
The model is trained end-to-end by minimizing the following objective function
H(Ŷ,Y) + γ||X̂−X||22, (4)
where H refers to the cross-entropy, Y to the true classification labels and γ is a tunable parameter to balance the supervised and the reconstruction losses.
2.2 FEATURE EMBEDDINGS
The feature embeddings used by the auxiliary networks can be either pre-computed or learnt offline, as well as learnt jointly with the rest of the architecture. In theory, any kind of embedding could be used, as long as we keep in mind that the goal is to reduce the number of free parameters of the basic model. In this work, we considered random projections (Bingham & Mannila, 2001), histograms (which are akin to bag-of-words representations), feature embeddings learnt offline (Mikolov et al., 2013) and feature embeddings jointly learnt with the rest of the proposed architecture.
Random projection: Randomly initializing an MLP defines a random projection. By using such a projection to encode the high-dimensional feature space into a more manageable lower-dimensional space, we were able to obtain decent results.
Per class histogram: For a given SNP, we can define a histogram of the values it can take over the whole population. Once normalized, this yields 3 values per SNP, corresponding to the proportion of the population having the values 0, 1 and 2 respectively for that SNP. After initial tests showed this was too coarse a representation for the dataset, we instead chose to consider the per-class proportion of the three values. With 26 classes in the 1000 Genomes dataset, this yields an embedding of size 78 for each feature. By this method, the matrix XT is summarized as a Nd × 78 matrix, where Nd is the number of SNPs in the dataset.
SNPtoVec: In Mikolov et al. (2013), the authors propose a word embedding that allows good reconstruction of the words’ context (surrounding words) by a neural network. SNPs do not have a similarly well-defined positional context (SNPs close together in our ordering might very well be independent) so our embedding is instead built by training a denoising autoencoder (DAE) (Vincent et al., 2008) on the matrix X. Thus, the DAE learns to recover the values of missing SNPs by leveraging their similarities and cooccurences with other SNPs. Once the DAE is trained, we obtain an encoding for each feature by feeding to the DAE an input where only that feature is active (the other features are set to 0s) and computing the hidden representation of the autoencoder for that single-feature input.
Embedding learnt end-to-end from raw data: In this case, we consider the feature embedding to be another MLP, whose input corresponds to the values that a SNP takes for each of the training samples and, whose parameters are learnt jointly with the rest of the network. Note that the layer(s) corresponding to the feature embedding are shared among auxiliary networks. For experiments reported in Section 4, we used a single hidden layer as embedding.
3 DATA: THE 1000 GENOMES PROJECT
The 1000 Genomes project is the first project to sequence the genomes of a large number of people in populations worldwide, yielding the largest public catalog of human genetic variants to date Consortium (2015). This allowed large-scale comparison of DNA sequences from populations, thanks to the presence of genetic variation. Individuals of the 1000 Genomes project are samples taken from 26 populations over the world, which are grouped into 5 geographical regions. Figure 2(a) shows a histogram derived from the 1000 Genomes data, depicting the frequency of individuals per population (ethnicity). Analogously, Figure 2(b) depicts the frequency of individuals per geographical region.
In this dataset, we included 315,345 genetic variants with frequencies of at least 5% in 3,450 individuals sampled worldwide from 26 populations, interrogated using microarray genotyping technology: the Genome-Wide Human SNP Array 6.0 by Affymetrix. The mutated state is established by comparison to the Genome Reference Consortium human genome (build 37). Since individuals have 2 copies of each genomic position, a sampled individual can have 0, 1 or 2 copies of a genetic mutation, hereafter referred to as an individual genotype. We excluded SNPs positioned on the sex
chromosomes and only included SNPs in approximate linkage equilibrium with each other, such that genotypes at neighboring positions are only weakly correlated (r2 < 0.5).
4 EXPERIMENTS
In this section, we describe the model architectures, and report and discuss the obtained results.
4.1 MODEL ARCHITECTURE
We experimented with simple models both in the auxiliary networks and the basic architecture, which yielded very promising results. We designed a basic architecture with 2 hidden layers fol-
lowed by a softmax layer to perform ancestry prediction. We trained this architecture with and without the assistance of the auxiliary network. Similarly, the auxiliary networks were build by stacking a hidden layer on top of one of the feature embeddings described in Section 2.2. In the reported experiments, all hidden layers have 100 units. All models were trained by means of stochastic gradient descent with adaptive learning rate (Tieleman & Hinton, 2012), both for γ = 0 and γ = 10, using dropout, limiting the norm of the weights to 1 and/or applying weight decay to reduce overfitting.2.
4.2 RESULTS
Given the relatively small amount of samples in the 1000 Genomes data, we report results obtained by 5-fold cross validation of the model. We split the data into 5 folds of equal size. A single fold is retained for test, whereas three of the remaining folds are used as training data and the final fold is used as validation data. We repeated the process 5 times (one per fold) and report the means and standard deviations of results on the different test sets.
Table 1 summarizes the results obtained for each model. First, we observe that, for most of Diet Network architectures, training with an reconstruction term in the loss (γ > 0) reduces the misclassification error and provides a lower standard deviation over the folds, suggesting more robustness to variations in the learnt feature embedding.
Training the models end-to-end, with no pre-computed feature embedding, yielded higher misclassification error than simply training the basic model, which could be attributed to the fact that adding the prediction networks makes a difficult, high-dimensional optimization problem even harder. As a general trend, adding pre-computed feature embeddings achieved better performance (lower error), while allowing to significantly reduce the number of free parameters in the fat layers of the model. Among the tested feature embeddings, random projections achieved good results, highlighting the potential of the model when reducing the number of free parameters.
Using the SNP2Vec embedding, trained to exploit the similarities and co-occurences between the SNPs, in conjunction with the Diet Networks framework obtains slightly better results than the model using a random projection. The addition of the reconstruction criterion does not appear to reduce the number of errors made by the model but it does appear to reduce the variance of the results, as observed on the other models.
Despite its simplicity, the per class histogram encoding (when used with a reconstruction criterion) yielded the best results. Note that this encoding is the one with the fewest number of free parameters in the fat layers, with a reduction factor of almost 4000 w.r.t. the analogous basic model (with reconstruction). Figure 3(a) shows the mean results obtained with the histogram embedding. As shown in the figure, when considering the ethnicity, the main misclassifications involve ethnicities likely to display very close genetic proximity, such as British from England and Scotland, and Utah residents with Northern and Western ancestry (likely to be immigrants from England), or Indian Telugu and Sri Lankan Tamil for instance. However, the model achieves almost 100% accuracy when considering the 5 geographical regions.
We also compared the performance of our model to the principal component analysis (PCA) approach, commonly used in the genomics domain, to select subgroups of individuals in order to perform more homogeneous analysis. The number of principal components (PCs) is chosen according to their significance, and usually varies from one dataset to another, being 10 the de facto standard for small datasets. However, in the case of the 1000 Genomes dataset, we could go up to 50 PCs. Therefore, we trained a linear classifier on top of PCA features, considering 10 and 50 PCs, 100 PCs to match the number of feature used in the other experiments, as well as 200 PCs. Using 200 PCs yielded better performance, but going beyond that saturated in terms of misclassification error (see SectionC in Appendix for more details). Adding hidden layers to the classifier didn’t help either (see reported results for several MLP configurations before the linear classifier).
5 CONCLUSION
In this paper, we proposed Diet Networks, a novel network parametrization, which considerably reduces the number of free parameters in the fat layers of a model when the input is very high
2The code to reproduce the experiments can be found here: https://github.com/adri-romsor/DietNetworks
dimensional. We showed how using the parameter prediction networks, yielded better generalization in terms of misclassification error. Notably, when using pre-computed feature embeddings that maximally reduced the number of free parameters, we were able to obtain our best results. We validated our approach on the publicly available 1000 genomes dataset, addressing the relevant task of ancestry prediction based on SNP data. This work demonstrated the potential of deep learning models to tackle domain-specific tasks where there is a mismatch between the number of samples and their high dimensionality.
Given the high accuracy achieved in the ancestry prediction task, we believe that deep learning techniques can improve standard practices in the analysis of human polymorphism data. We expect that these techniques will allow us to tackle the more challenging problem of conducting genetic association studies. Hence, we expect to further develop our method to conduct population-aware analyses of SNP data in disease cohorts. The increased power of deep learning methods to identify the genetic basis of common diseases could lead to better patient risk prediction and will improve our overall understanding of disease etiology.
ACKNOWLEDGMENTS
The authors would like to thank the developers of Theano Theano Development Team (2016) and Lasagne Lasagne (2016). We acknowledge the support of the following agencies for research funding and computing support: Imagia, CIFAR, Canada Research Chairs, Compute Canada and Calcul Québec. J.G.H. is an EPAC/Linacre Junior Research Fellow funded by the Human Frontiers Program (LT-001017/2013-L). Special thanks to Valèria Romero-Soriano, Xavier Grau-Bové and Margaux Luck for their patience sharing genomic biology expertise; as well as to Michal Drozdzal, Caglar Gulcehre and Simon Jégou for useful discussions and support.
A THE 1000 GENOMES PROJECT LEGENDS
A.1 POPULATION ETHNICITY LEGEND
ACB: African Caribbeans in Barbados ASW: Americans of African Ancestry in SW USA BEB: Bengali from Bangladesh CDX: Chinese Dai in Xishuangbanna CEU: Utah Residents (CEPH) with Northern and Western Ancestry CHB: Han Chinese in Bejing CHS: Southern Han Chinese CLM: Colombians from Medellin ESN: Esan in Nigeria FIN: Finnish in Finland GBR: British in England and Scotland GIH: Gujarati Indian from Houston GWD: Gambian in Western Divisions in the Gambia IBS: Iberian Population in Spain ITU: Indian Telugu from the UK JPT: Japanese in Tokyo KHV: Kinh in Ho Chi Minh City LWK: Luhya in Webuye MSL: Mende in Sierra Leone MXL: Mexican Ancestry from Los Angeles PEL: Peruvians from Lima PJL: Punjabi from Lahore PUR: Puerto Ricans STU: Sri Lankan Tamil from the UK TSI: Toscani in Italia YRI: Yoruba in Ibadan
A.2 GEOGPRAHICAL REGION LEGEND
AFR: African AMR: Ad Mixed American EAS: East Asian EUR: European SAS: South Asian
B OBTAINING THE 1000 GENOMES DATA
SNP data for the 1000G dataset was downloaded from ftp://ftp.1000genomes.ebi.ac. uk:21/vol1/ftp/release/20130502/supporting/hd_genotype_chip/
− ALL . wgs . n h g r i c o r i e l l a f f y 6 . 2 0 1 4 0 8 2 5 . g e n o t y p e s h a s p e d . v c f . gz
− a f f y s a m p l e s . 2 0 1 4 1 1 1 8 . p a n e l
Representative commands:
With PLINK v1.90b2n 64-bit https://www.cog-genomics.org/plink2
# c o n v e r t v c f t o p l i n k f o r m a t ( bed ) , and o n l y k e e p i n g common markers ( minor a l l e l e f r e q u e n c y > 0 . 0 5 i n combined sample ) > p l i n k −−v c f $ p a t h /ALL . wgs . n h g r i c o r i e l l a f f y 6 . 2 0 1 4 0 8 2 5 . g e n o t y p e s h a s p e d . v c f . gz −−maf 0 . 0 5 −−o u t $ p a t h / a f f y 6 b i a l l e l i c s n p s m a f 0 0 5 a u t −−not−c h r X Y MT −−make−bed
# produce a pruned s u b s e t o f markers t h a t are i n a p p r o x i m a t e l i n k a g e e q u i l i b r i u m w i t h each o t h e r > p l i n k −−b f i l e $ p a t h / a f f y 6 b i a l l e l i c s n p s m a f 0 0 5 a u t −−indep− p a i r w i s e 50 5 0 . 5 −−o u t $ p a t h / a f f y 6 b i a l l e l i c s n p s m a f 0 0 5 a u t
# e x c l u d e markers t o g e t pruned s u b s e t > p l i n k −−b f i l e $ p a t h / a f f y 6 b i a l l e l i c s n p s m a f 0 0 5 a u t −−e x c l u d e
$ p a t h / a f f y 6 b i a l l e l i c s n p s m a f 0 0 5 a u t . p rune . o u t −−r e c o d e A −− o u t $ p a t h / a f f y 6 b i a l l e l i c s n p s m a f 0 0 5 t h i n n e d a u t A
For information on how to download this pre-processed dataset directly, please email Adriana Romero or Pierre Luc Carrier.
C PCA COMPONENTS AND MISCLASSIFICATION ERROR
In this section, we analyze the influence of increasing the number of PCs used to perform classification. Figure 4 depicts the obtained results when considering 100, 200, 400, 800 and 1000 PCs. As shown in the figure, the best validation error comes with PC200. Further increasing the number of PCs improves the training error but does not generalize well on the validation set.

The paper presents an application of deep learning to genomic SNP data
with a comparison of possible approaches for dealing with the very
high data dimensionality. The approach looks very interesting but the
experiments are too limited to draw firm conclusions about the
strengths of different approaches. The presentation would benefit from
more precise math.
Quality:
The basic idea of the paper is interesting and the applied deep
learning methodology appears reasonable. The experimental evaluation
is rather weak as it only covers a single data set and a very limited
number of cross validation folds. Given the significant variation in
the performances of all the methods, it seems the differences between
the better-performing methods are probably not statistically
significant. More comprehensive empirical validation could clearly
strengthen the paper.
Clarity:
The writing is generally good both in terms of the biology and ML, but
more mathematical rigour would make it easier to understand precisely
what was done. The different architectures are explained on an
intuitive level and might benefit from a clear mathematical
definition. I was ultimately left unsure of what the "raw end2end"
model is - given so few parameters it cannot work on raw 300k
dimensional input but I could not figure out what kind of embedding
was used.
The results in Fig. 3 might be clearer if scaled so that maximum for
each class is 1 to avoid confounding from different numbers of
subjects in different classes. In the text, please use the standard
italics math font for all symbols such as N, N_d, ...
Originality:
The application and the approach appear quite novel.
Significance:
There is clearly strong interest for deep learning in the genomics
area and the paper seeks to address some of the major bottlenecks
here. It is too early to tell whether the specific techniques proposed
in the paper will be the ultimate solution, but at the very least the
paper provides interesting new ideas for others to work on.
Other comments:
I think releasing the code as promised would be a must.