TREE-STRUCTURED DECODING WITH DOUBLY- RECURRENT NEURAL NETWORKS
We propose a neural network architecture for generating tree-structured objects from encoded representations. The core of the method is a doubly recurrent neural network model comprised of separate width and depth recurrences that are combined inside each cell (node) to generate an output. The topology of the tree is modeled explicitly together with the content. That is, in response to an encoded vector representation, co-evolving recurrences are used to realize the associated tree and the labels for the nodes in the tree. We test this architecture in an encoderdecoder framework, where we train a network to encode a sentence as a vector, and then generate a tree structure from it. The experimental results show the effectiveness of this architecture at recovering latent tree structure in sequences and at mapping sentences to simple functional programs.
We propose a neural network architecture for generating tree-structured objects from encoded representations. The core of the method is a doubly recurrent neural network model comprised of separate width and depth recurrences that are combined inside each cell (node) to generate an output. The topology of the tree is modeled explicitly together with the content. That is, in response to an encoded vector representation, co-evolving recurrences are used to realize the associated tree and the labels for the nodes in the tree. We test this architecture in an encoderdecoder framework, where we train a network to encode a sentence as a vector, and then generate a tree structure from it. The experimental results show the effectiveness of this architecture at recovering latent tree structure in sequences and at mapping sentences to simple functional programs.
1 INTRODUCTION
Recurrent neural networks have become extremely popular for modeling structured data. Key to their success is their ability to learn long-range temporal dependencies, their flexibility, and ease of customization. These architectures are naturally suited for modeling sequences since the underlying state evolution resulting from successive operations follows an inherently linear order (Williams & Zipser, 1995; Hochreiter & Schmidhuber, 1997). Indeed, they have been successfully adapted to language modeling (Zaremba et al., 2015), machine translation (Sutskever et al., 2014) and conversational agents (Vinyals & Le, 2015), among other applications.
Although sequences arise frequently in practice, other structures such as trees or graphs do not naturally conform to a linear ordering. For example, natural language sentences or associated parse trees, programs, hierarchical structures in biology, or molecules are not inherently linear structures. While sentences in natural language can be modeled as if they were linear sequences, the underlying process is compositional (Frege, 1892). Models that construct sentences compositionally should derive an advantage from adopting a more appropriate inductive bias.
The flexibility and success of recurrent neural networks in modeling and generating sequential data has prompted efforts to adapt them to non-sequential data too. Recent work has focused on the application of neural architectures to hierarchical structures, albeit in limited ways. Much of this work has assumed that either the full tree structure is given (Socher et al., 2012; Tai et al., 2015) or at least the nodes are (Socher & Lin, 2011; Chen & Manning, 2014; Kiperwasser & Goldberg, 2016). In the former scenario, the network aggregates the node information in a manner that is coherent with a given tree structure while, in the latter, generation is reduced to an attachment problem, i.e., sequentially deciding which pairs of nodes to join with an edge until a tree is formed.
The full problem of decoding with structure, i.e., generating a tree-structured object with node labels from a given vector representation, has remained largely unexplored until recently. Recent efforts to adapt RNNs to this context have so far remained relatively close to their sequential counterparts. For example, in order to capture depth and branching in the tree, one can introduce special tokens (Dong & Lapata, 2016) or use alternating RNNs coupled with external classifiers to predict branching (Zhang et al., 2016).
In this work, we propose a novel architecture tailored specifically to tree-structured decoding. At the heart of our approach is a doubly-recurrent (breadth and depth-wise recurrent) neural network which separately models the flow of information between parent and children nodes, and between siblings. Each of these relationships is modeled with a recurrent module whose hidden states are updated upon observing node labels. Every node in the tree receives two hidden states, which are then combined and used to predict a label for that node. Besides maintaining separate but simultaneous fraternal and paternal recurrences, the proposed architecture departs from previous methods in that it explicitly models tree topology. Each node in the network has modules that predict, based on the cell state, whether the node is terminal, both in terms of depth and width. Decoupling these decisions from the label prediction allows for a more concise formulation, which does not require artificial tokens to be added to the tree to simulate branching.
We test this novel architecture in various encoder-decoder frameworks, coupling it with sequential encoders to predict tree structure from encoded vector representations of sequences. The experimental results show the effectiveness of this approach at recovering latent structure in flattened string representations of trees (Section 4.1) and at mapping from natural language descriptions of simple programs to abstract syntax trees (Section 4.2). In addition, we show that even for sequence-tosequence tasks such as machine translation, the proposed architecture exhibits desirable properties, such as invariance to structural changes and coarse-to-fine generation (Section 4.3).
To summarize, the main contributions of this paper are as follows:
• We propose a novel neural network architecture specifically tailored to tree-structured decoding, which maintains separate depth and width recurrent states and combines them to obtain hidden states for every node in the tree.
• We equip this novel architecture with a mechanism to predict tree topology explicitly (as opposed to implicitly by adding nodes with special tokens).
• We show experimentally that the proposed method is capable of recovering trees from encoded representations and that it outperforms state-of-the-art methods in a task consisting of mapping sentences to simple functional programs.
2 RELATED WORK
Recursive Neural Networks. Recursive neural networks (Socher & Lin, 2011; Socher et al., 2012) were proposed to model data with hierarchical structures, such as parsed scenes and natural language sentences. Though they have been most successfully applied to encoding objects when their treestructured representation is given (Socher et al., 2013), the original formulation by Socher & Lin (2011) also considered using them to predict the structure (edges), albeit for the case where nodes are given. Thus, besides their limited applicability due to their assumption of binary trees, recursive neural networks are not useful for fully generating trees from scratch.
Tree-structured encoders. The Tree-LSTM of Tai et al. (2015) is a generalization of long shortterm memory networks (Hochreiter & Schmidhuber, 1997) to tree-structured inputs. Their model constructs a sentence representation bottom-up, obtaining at every step the representation of a node in the tree from those of its children. In this sense, this model can be seen as a generalization of recursive neural networks to trees with degree potentially greater than two, with the additional longrange dependency modeling provided by LSTMs. They propose two methods for aggregating the states of the children, depending on the type of underlying tree: N-ary trees or trees with unknown and potentially unbounded branching factor. TreeLSTMs have shown promising results for compositional encoding of structured data, though by construction they cannot be used for decoding, since they operate on a given tree structure.
Tree-structured decoders. Proposed only very recently, most tree-structured decoders rely on stacked on intertwined RNNs, and use heuristic methods for topological decisions during generation. Closest to our method is the Top-down Tree LSTM of Zhang et al. (2016), which generates a tree from an encoded representation. Their method relies on 4 independent LSTMs, which act in alternation—as opposed to simultaneously in our approach—yielding essentially a standard LSTM that changes the weights it uses based on the position of the current node. In addition, their method
provides children with asymmetric parent input: “younger” children receive information from the parent state only through the previous sibling’s state. Though most of their experiments focus on the case where the nodes are given, they mention how to use their method for full prediction by introducing additional binary classifiers which predict which of the four LSTMs is to be used. These classifiers are trained in isolation after the main architecture has been trained. Contrary to this approach, our method can be trained end-to-end in only one pass, has a simpler formulation and explicitly incorporates topological prediction as part of the functioning of each neuron.
A similar approach is proposed by Dong & Lapata (2016). They propose SEQ2TREE, an encoderdecoder architecture that maps sentences to tree structures. For the decoder, they rely on hierarchical use of an LSTM, similar to Tai et al. (2015), but in the opposite direction: working top-down from the root of the tree. To decide when to change levels in the hierarchy, they augment the training trees with nonterminal nodes labeled with a special token <n>, which when generated during decoding trigger the branching out into a lower level in the tree. Similar to our method, they feed nodes with hidden representations of their parent and sibling, but they do so by concatenating both states and running them through a single recurrent unit, as opposed to our method, where these two sources of information are handled separately. A further difference is that our approach does not require artificial nodes with special tokens to be added to the tree, resulting in smaller trees.
Hierarchical Neural Networks for Parsing. Neural networks have also been recently introduced to the problem of natural language parsing (Chen & Manning, 2014; Kiperwasser & Goldberg, 2016). In this problem, the task is to predict a parse tree over a given sentence. For this, Kiperwasser & Goldberg (2016) use recurrent neural networks as a building block, and compose them recursively to obtain a tree-structured encoder. Starting from the leaves (words) they predict a parse tree with a projective bottom-up strategy, which sequentially updates the encoded vector representation of the tree and uses it to guide edge-attaching decisions. Though conceptually similar to our approach, their method relies on having access to the nodes of the tree (words) and only predicts its topology, so—similar to recursive neural networks—it cannot be used for a fully generative decoding.
3 DOUBLY RECURRENT NEURAL NETWORKS
Generating a tree-structured object from scratch using only an encoded representation poses several design challenges. First, one must decide in which order to generate the tree. If the nodes on the decoder side were given (such as in parsing), it would be possible to generate a tree bottom-up from these nodes (e.g. as Kiperwasser & Goldberg 2016 do). In the setting we are interested in, however, not even the nodes are known when decoding, so the natural choice is a top-down decoder, which starting from an encoded representation generates the root of the tree and then recursively generates the children (if any) of every node.
The second challenge arises from the asymmetric hierarchical nature of trees. Unlike the sequenceto-sequence setting where encoding and decoding can be achieved with analogous procedures, when dealing with tree-structured data these two involve significantly different operations. For example, an encoder that processes a tree bottom-up using information of a node’s children to obtain its representation cannot be simply reversed and used as a decoder, since when generating the tree top-down, nodes have to be generated before their children are.
An additional design constraint comes from deciding what information to feed to each node. For sequences, the choice is obvious: a node should receive information from the node preceding or succeeding it (or both), i.e. there is a one-dimensional flow of information. In trees, there is an evident flow of information from parent to children (or vice-versa), but when generating nodes in a top-down order it seems unnatural to generate children in isolation: the label of one of them will likely influence what the states of the other children might be. For example, in the case of parse trees, generating a verb will reduce the chances of other verbs occurring in that branch.
With these considerations in mind, we propose an architecture tailored to tree decoding from scratch: top-down, recursive and doubly-recurrent, i.e. where both the ancestral (parent-to-children) and fraternal (sibling-to-sibling) flows of information are modeled with recurrent modules. Thus, the building block of a doubly recurrent neural network (DRNN) is a cell with two types of input states, one coming from its parent, updated and passed on to its descendants, and another one received from
its previous sibling,1 updated and passed on to the next one. We model the flow of information in the two directions with separate recurrent modules.
Formally, let T = {V, E ,X} be a connected labeled tree, where V is the set of nodes, E the set of edges and X are node labels.2 Let ga and gf be functions which apply one step of the two separate RNNs. For a node i ∈ V with parent p(i) and previous sibling s(i), the ancestral and fraternal hidden states are updated via
hai = g a(hap(i),xp(i)) (1)
hfi = g f (hfs(i),xs(i)) (2)
where xs(j),xp(i) are the vectors representing the previous sibling’s and parent’s values, respectively. Once the hidden depth and width states have been updated with these observed labels, they are combined to obtain a predictive hidden state:
h (pred) i = tanh ( Ufhfi +U ahai ) (3)
where Uf ∈ Rn×Df and Ua ∈ Rn×Da are learnable parameters. This state contains combined information of the node’s neighborhood in the tree, and is used to predict a label for it. In its simplest form, the network could compute the output of node i by sampling from distribution
oi = softmax(Wh (pred) i ) (4)
In the next section, we propose a slight modification to (4) whereby topological information is included in the computation of cell outputs. After the node’s output symbol xi has been obtained by sampling from oi, the cell passes hai to all its children and h f i to the next sibling (if any), enabling them to apply Eqs (1) and (2) to realize their states. This procedure continues recursively, until termination conditions (explained in the next section) cause it to halt.
3.1 TOPOLOGICAL PREDICTION
As mentioned before, the central issue with free-form tree construction is to predict the topology of the tree. When constructing the tree top-down, for each node we need to decide: (i) whether it is a leaf node (and thus it should not produce offspring) and (ii) whether there should be additional siblings produced after it. Answering these two questions for every node allows us to construct a tree from scratch and eventual stop growing it.
Sequence decoders typically rely on special tokens to terminate generation (Sutskever et al., 2014). The token is added to the vocabulary and treated as a regular word. During training, the examples are padded with this token at the end of the sequence, and during testing, generation of this token signals termination. These ideas has been adopted by most tree decoders (Dong & Lapata, 2016). There are two important downsides of using a padding strategy for topology prediction in trees. First, the size of the tree can grow considerably. While in the sequence framework only one stopping token is needed, a tree with n nodes might need up to O(n) padding nodes to be added. This can have important effects in training speed. The second reason is that a single stopping token selected competitively with other tokens requires one to continually update the associated parameters in response to any changes in the distribution over ordinary tokens so as to maintain topological control.
Based on these observations, we propose an alternative approach to stopping, in which topological decisions are made explicitly (as opposed to implicitly, with stopping tokens). For this, we use the predictive hidden state of the node h(pred) with a projection and sigmoid activation:
pai = σ(u a · h(pred)i ) (5)
The value pai ∈ [0, 1] is interpreted as the probability that node i has children. Analogously, we can obtain a probability of stopping fraternal branch growth after the current node as follows:
pfi = σ(u f · h(pred)i ) (6)
1Unlike the “ancestral” line, the order within sibling nodes is ambiguous. While in abstract trees it is assumed that the there is no such ordering, we assume that for the structures were are interested in learning there is always one: either chronological (the temporal order in which the nodes were generated) or latent (e.g. the grammatical order of the words in a parse tree with respect to their sentence representation).
2We assume throughout that these values are given as class indicators xi ∈ {1, . . . , N}.
Note that these stopping strategies depart from the usual padding methods in a fundamental property: the decision to stop is made before instead of in conjunction with the label prediction. The rationale behind this is that the label of a node will likely be influenced not only by its context, but also by the type of node (terminal or non-terminal) where it is to be assigned. This is the case in language, for example, where syntactic constraints restrict the type of words that can be found in terminal nodes. For this purpose, we include the topological information as inputs to the label prediction layer. Thus, (4) takes the form
oi = softmax(Wh (pred) i + αiv a + ϕiv f ) (7)
where αi, ϕi ∈ {0, 1} are binary variables indicating the topological decisions and va,vf are learnable offset parameters. During training, we use gold-truth values in (7), i.e. αi = 1 if node i has children and ϕi = 1 if it has a succeeding sibling. During testing, these values are obtained from pa, pf by sampling or beam-search. A schematic representation of the internal structure of a DRNN cell and the flow of information in a tree are shown in Figure 1.
3.2 TRAINING DRNNS
We train DRNNs with (reverse) back-propagation through structure (BPTS) (Goller & Kuechler, 1996). In the forward pass, node outputs are computed in a top-down fashion on the structureunrolled version of the network, following the natural3 dependencies of the tree. We obtain error signal at the node level from the two types of prediction: label and topology. For the former, we compute cross-entropy loss of oi with respect to the true label of the node xi. For the topological values pai and p f i we compute binary cross entropy loss with respect to gold topological indicators αi, ϕi ∈ {0, 1}. In the backward pass, we proceed in the reverse (bottom-up) direction, feeding into every node the gradients received from child and sibling nodes and computing internally gradients with respect to both topology and label prediction. Further details on the backpropagation flow are provided in the Appendix.
Note that the way BPTS is computed implies and underlying decoupled loss function L(x̂) = ∑ i∈V Llabel(xi, x̂i) + Ltopo(pi, p̂i) (8)
The decoupled nature of this loss allows us to weigh these two objectives differently, to emphasize either topology or label prediction accuracy. Investigating the effect of this is left for future work.
3The traversal is always breadth-first starting from the root, but the order in which sibling nodes are visited might depend on the specific problem. If the nodes of the tree have an underlying order (such as in dependency parse trees), it is usually desirable to preserve this order.
As is common with sequence generation, during training we perform teacher forcing: after predicting the label of a node and its corresponding loss, we replace it with its gold value, so that children and siblings receive the correct label for that node. Analogously, we obtain the probabilities pa and pf , compute their loss, and replace them for ground truth variables αi, ϕi for all downstream computations. Addressing this exposure bias by mixing ground truth labels with model predictions during training (Venkatraman et al., 2015) or by incremental hybrid losses (Ranzato et al., 2016) is left as an avenue for future work.
4 EXPERIMENTS
4.1 SYNTHETIC TREE RECOVERY
In our first set of experiments we evaluate the effectiveness of the proposed architecture to recover trees from flattened string representations. For this, we first generate a toy dataset consisting of simple labeled trees. To isolate the effect of label content from topological prediction, we take a small vocabulary consisting of the 26 letters of the English alphabet. We generate trees in a top-down fashion, conditioning the label and topology of every node on the state of its ancestors and siblings. For simplicity, we use a Markovian assumption on these dependencies, modeling the probability of a node’s label as depending only on the label of its parent and the last sibling generated before it (if any). Conditioned on these two inputs, we model the label of the node as coming from a multinomial distribution over the alphabet with a dirichlet prior. To generate the topology of the tree, we model the probability of a node having children and a next-sibling as depending only on its label and the depth of the tree. For each tree we generate a string representation by traversing it in breadth-first preorder, starting from the root. The labels of the nodes are concatenated into a string in the order in which they were visited, resulting in a string of |T | symbols. We create a dataset of 5,000 trees with this procedure, and split it randomly into train, validation and test sets (with a 80%,10%,10% split). Further details on the construction of this dataset are provided in the Appendix.
The task consists of learning a mapping from strings to trees, and using this learned mapping to recover the tree structure of the test set examples, given only their flattened representation. To do so, we use an encoder-decoder framework, where the strings are mapped to a fixed-size vector representation using a recurrent neural network. For the decoder, we use a DRNN with LSTM modules, which given the encoded representation generates a tree. We choose hyper-parameters with cross-validation. Full training details are provided in the Appendix.
Measuring performance only in terms of exact recovery would likely yield near-zero accuracies for most trees. Instead, we opt for a finer-grained metric of tree similarity that gives partial credit for correctly predicted subtrees. Treating tree generation as a retrieval problem, we evaluate the quality of the predicted tree in terms of the precision and recall of recovering nodes and edges present in the gold tree. Thus, we penalize both missing and superfluous components. As baseline, we induce a probabilistic context-free grammar (PCFG) on the full training data and use it to parse the test sentences. Note that unlike the DRNN, this parser has direct access to the sentence representation and thus its task is only to infer the tree structure on top of it, so this is indeed a strong baseline.
Figure 3 shows the results on the test set. Training on the full data yields node and edge retrieval F1-Scores of 75% and 71%, respectively, the latter considerably above the baseline.4 This 4% gap can be explained by correct nodes being generated in the wrong part of the tree, as in the example in
4Since the PCFG parser has access to the nodes by construction, node accuracy for the baseline method is irrelevant and thus omitted from the analysis.
Figure 2. The second plot in Figure 3 shows that although small trees are recovered more accurately, precision decays slowly with tree size, with depth accounting for the largest effect (Figure 4).
4.2 MAPPING SENTENCES TO FUNCTIONAL PROGRAMS
Tree structures arise naturally in the context of programs. A typical compiler takes human-readable source code (expressed as sequences of characters) and transforms it into an executable abstract syntax tree (AST). Source code, however, is already semi-structured. Mapping natural language sentences directly into executable programs is an open problem, which has received considerable interest in the natural language processing community (Kate et al., 2005; Branavan et al., 2009).
The IFTTT dataset (Quirk et al., 2015) is a simple testbed for language-to-program mapping. It consists of if-this-then-that programs (called recipes) crawled from the IFTTT website5, paired with natural language descriptions of their purpose. The recipes consist of a trigger and an action, each defined in terms of a channel (e.g. “Facebook”), a function (e.g. “Post a status update”) and potentially arguments and parameters. An example of a recipe and its description are shown in Figure 5. The data is user-generated and extremely noisy, which makes the task significantly challenging.
5www.ifttt.com
We approach this task using an encoder-decoder framework. We use a standard RNN encoder, either an LSTM or a GRU (Cho et al., 2014), to map the sentence to a vector representation, and we use a DRNN decoder to generate the AST representation of the recipe. We use the original data split, which consists of 77,495 training, 5,171 development and 4,294 test examples. For evaluation, we use the same metrics as Quirk et al. (2015), who note that computing exact accuracy on such a noisy dataset is problematic, and instead propose to evaluate the generated AST in terms of F1-score on the set of recovered productions. In addition, they compute accuracy at the channel level (i.e. when both channels are predicted correctly) and at the function level (both channels and both functions predicted correctly).
We compare our methods against the various extraction and phrased-based machine translation baselines of Quirk et al. (2015) and the the methods of Dong & Lapata (2016): SEQ2SEQ, a sequenceto-sequence model trained on flattened representations of the AST, and SEQ2TREE, a token-driven hierarchical RNN. Following these two works, we report results on two noise-filtered subsets of the data: one with all non-English and unintelligible recipes removed and the other one with recipes for which at least three humans agreed with the gold AST. The results are shown in Table 1. In both subsets, DRNNs perform on par or above previous approaches, with LSTM-DRNN achieving significantly better results. The improvement is particularly evident in terms of F1-score, which is the only metric used by previous approaches that measures global tree reconstruction accuracy. To better understand the quality of the predicted trees beyond the function level (i.e. (b) in Figure 5), we computed node accuracy on the arguments level. Our best performing model, LSTM-DRNN, achieves a Macro F1 score of 51% (0.71 precision, 0.40 recall) over argument nodes, which shows that the model is reasonably successful at predicting structure even beyond depth three. The best performing alternative model, SEQ2TREE, achieves a corresponding F1 score of 46%.
4.3 MACHINE TRANSLATION
In our last set of experiments, we offer a qualitative evaluation DRNNs in the context of machine translation. Obtaining state-of-the-art results in machine translation requires highly-optimized architectures and large parallel corpora. This is not our goal. Instead, we investigate whether decoding with structure can bring benefits to a task traditionally approached as a sequence-to-sequence problem. For this reason, we consider a setting with limited data: a subset of the WMT14 dataset consisting of about 50K English↔ French sentence pairs (see the Appendix for details) along with dependency parses of the target (English) side.
We train a sequence-to-tree model using an LSTM encoder and a DRNN decoder as in the previous experiments. A slight modification here is that we distinguish left and right children in the tree, using two symmetric width-modules gfL, g f R that produce children from the parent outwards. With this, children are lexically ordered, and therefore trees can be easily and un-ambiguously projected back into sentences. We compare our model against a sequence-to-sequence architecture of similar complexity (in terms of number of parameters) trained on the same data using the optimized OpenNMT library (Klein et al., 2017). For decoding, we use a simple best-of-k sampling scheme for our model, and beam search for the SEQ2SEQ models.
Source “ produit différentes réponses qui changent avec le temps selon nos expériences et nos relations ”
“je ne sais jamais quoi dire dans ces cas là”
SEQ2SEQ: l = 1 a I l = 4 with the different actions I do l = 8 with the different actions who change with I do not know what to say
DRNN: d = 1 answers know d = 2 different answers change but i do not know d = 3 product the different answers change . but i do not know to say
Table 2: Translations at different resolutions (size constraints imposed during decoding) for two example sentences.
First, we analyze the quality of translations as a function of the maximum allowed target sentence “size”. The notion of size for a sequence decoder is simply the length while for DRNN we use depth instead so as to tap into the inherent granularity at which sentences can be generated from this architecture. Two such examples are shown in Table 2. Since DRNN topology has been trained to mimic dependency parses top-down, the decoder tends to first generate the fundamental aspects of the sentence (verb, nouns), leaving less important refinements for deeper structures down in the tree. The sequence decoder, in contrast, is trained for left-to-right sequential generation, and thus produces less informative translations under max-length restrictions.
In our second experiment we investigate the decoders’ ability to entertain natural paraphrases of sentences. If we keep the semantic content of a sentence fixed and only change its grammatical structure, it is desirable that the decoder would assign nearly the same likelihood to the new sentence. One way to assess this invariance is to compare the relative likelihood that the model assigns to the gold sentence in comparison to its paraphrase. To test this, we take 50 examples from the WMT test split and manually generate paraphrases with various types of structural alterations (see details in the Appendix). For each type of decoder, we measure the relative change (in absolute value) of the log-likelihood resulting from the perturbation. All the models we compare have similar standard deviation (40 ± 20) of log-likelihood scores over these examples, so the relative changes in the log-likelihood remain directly comparable. For each architecture we train two versions of different sizes, where the sizes are balanced in terms of the number of parameters across the architectures. The results in Figure 6 show that DRNN’s exhibit significantly lower log-likelihood change, suggesting that, as language models, they are more robust to natural structural variation than their SEQ2SEQ counterparts.
5 DISCUSSION AND FUTURE WORK
We have presented doubly recurrent neural networks, a natural extension of (sequential) recurrent architectures to tree-structured objects. This architecture models the information flow in a tree with two separate recurrent modules: one carrying ancestral information (received from parent and passed on to offspring) and the other carrying fraternal information (passed from sibling to sibling). The topology of the tree is modeled explicitly and separately from the label prediction, with modules that given the state of a node predict whether it has children and siblings.
The experimental results show that the proposed method is able to predict reasonable tree structures from encoded vector representations. Despite the simple structure of the IFTTT trees, the results on that task suggest a promising direction of using DRNNs for generating programs or executable queries from natural language. On the other hand, the results on the toy machine translation task show that even when used to generate sequences, DRNN’s exhibit desirable properties, such as invariance over structural modifications and the ability to perform coarse-to-fine decoding. In order to truly use this architecture for machine translation, the approach must be scaled by resorting to batch processing in GPU. This is possible since forward and backward propagation are computed sequentially along tree traversal paths so that inputs and hidden states of parents and siblings can be grouped into tensors and operated in batch. We leave this as an avenue for future work.
ACKNOWLEDGEMENTS
DA-M acknowledges support from a CONACYT fellowship. The authors would like to thank the anonymous reviewers for their constructive comments.
B TRAINING DETAILS
B.1 BACKPROPAGATION WITH DRNN’S
During training, we do the forward pass over the trees in breadth-first preorder, feeding into every node an ancestral and a fraternal state. For computational efficiency, before passing on the ancestral state to the offspring, we update it through the RNN using the current node’s label, so as to avoid repeating this step for every child node. After the forward pass is complete, we compute label (cross-entropy) and topological (binary cross-entropy) loss for every node. In the backward pass, we compute in this order:
1. Gradient of the current node’s label prediction loss with respect to softmax layer parameters W,va,vf : ∇θL(xi, x̂i).
2. Gradients of topological prediction variable loss with respect to sigmoid layer parameters: ∇θL(pai , tai ) and ∇θL(pfi , tfi ).
3. Gradient of predictive state layer parameters with respect to h(pred). 4. Gradient of predicted ancestral and fraternal hidden states with respect to gf and ga’s pa-
rameters.
The gradients of the input ancestral and fraternal hidden states are then passed on to the previous sibling and parent. When nodes have more than one child, we combine gradients from multiple children by averaging them. This procedure is repeated until the root note is reached, after which a single (ancestral state) gradient is passed to the encoder.
B.2 MODEL SPECIFICATION AND TRAINING PARAMETERS
The best parameters for all tasks are chosen by performance on the validation sets. We perform early stopping based on the validation loss. For the IFTTT task, we initialize word embeddings with pretrained GloVe vectors (Pennington et al., 2014). For both tasks we clip gradients when the absolute value of any element exceeds 5. We regularize with a small penalty ρ on the l2 norm of the parameters. We train all methods with ADAM (Kingma & Ba, 2014), with initial learning rate chosen by cross-validation. The parameter configurations that yielded the best results and were used for the final models are shown in Table 3. Details about the four models used for the machine translation task are shown in Table 4.
C DATASET DETAILS
C.1 SYNTHETIC TREE DATASET GENERATION
We generate trees in a top-down fashion, conditioning the label and topology of every node on the state of its ancestors and siblings. For simplicity, we use a Markovian assumption on these dependencies, modeling the probability of a node’s label as depending only on the label of its parent p(i) and the last sibling s(i) generated before it (if any). Conditioned on these two inputs, we model the label of the node as coming from a multinomial distribution over the alphabet:
P (wi | T ) = P (w | wp(i), ws(i)) ∼ Multi(θwp(i),ws(i)) (9) where θwp(i),ws(i) are class probabilities drawn from a Dirichlet prior with parameter αv . On the other hand, we denote by bai the binary variable indicating whether node i has descendants, and by bfi that indicating whether it has an ensuing sibling. We model these variables as depending only on the label of the current node and its position in the tree:
P (bai | T ) = P (bai | wi, Di) = Bernoulli(pawi · ga(Di)) P (bfi | T ) = P (bfi | wi,Wi) = Bernoulli(pfwi · gf (Wi))
whereDi is the depth of node i andWi its width, defined as its position among the children of its parent p(i). Intuitively, we want to make P (bai = 1 | T ) decrease as we go deeper and further along the branches of the tree, so as to control its growth. Thus, we model ga and gf as decreasing functions with geometric decay, namely ga(D) = (γa)D and gf (W ) = (γf )W , with γa, γf ∈ (0, 1). For the label-conditioned branching probabilities P (bai | wi) and P (bfi | wi), we use Bernoulli distributions with probabilities drawn from beta priors with parameters (αa, βa) and (αf , βf ), respectively.
In summary, we use the following generative procedure to grow the trees:
1. For each wi ∈ V , draw pawi ∼ Beta(αa, βa) and pfwi ∼ Beta(αf , βf ) 2. For each pair (wi, wj) draw θwi,wj ∼ Dir(αV ) 3. While there is an unlabeled non-terminal node i do:
• Sample a label for i from w∗ ∼ P (w|wp(i), ws(i)) = Multi(θwp(i),ws(i)). • Draw ba ∼ P (ba|w∗, D) = Bernoulli(γDa · paw(i)), where D is the current depth. If ba = 1, generate an node k, set p(k) = i, and add it to the queue.
• Draw ba ∼ P (bf |w∗, D) = Bernoulli(γWf · pfw(i)), where W is the current width. If bf = 1, generate an node k, set s(k) = i, and add it to the queue.
Note that this generative process does create a dependence between the topology and content of the trees (since the variables ba and bf depend on the content of the tree via their dependence on the label of their corresponding node). However, the actual process by which labels and topological decision is generated relies on separate mechanisms. This is natural assumption which is reasonable to expect in practice.
The choice of prior parameters is done drawing inspiration from natural language parse trees. We want nodes to have low but diverse probabilities of generating children, so we seek a slow-decaying distribution with most mass allocated in values close to 0. For this, we use (αa, βa) = (0.25, 1). For sibling generation, we use (αf , βf ) = (7, 2), which yields a distribution concentrated in values close to 1, so that nodes have on average a high and similar probability of producing siblings. Since we seek trees that are wider than they are deep, we use decay parameters γa = 0.6, γf = 0.9. Finally, we use a αv = 10 · 1 for the parent-sibling probability prior, favoring non-uniform interactions. Using this configuration, we generate 5000 sentence-tree pairs, which we split into training (4000 examples), validation (500) and test (500) sets. The characteristics of the trees in the dataset are summarized in Table 5.
The IFTTT dataset comes with a script to generate the data by crawling and parsing the recipes. Unfortunately, by the time we ran the script many recipes had been removed or changed. We therefore resorted to the original dataset used by Quirk et al. (2015). We converted these recipes into our tree format, assigning a node to each element in the first three levels (channels, functions and arguments, see figure 5). For the parameters level, many recipes have sentences instead of single tokens, so we broke these up creating one node per word. The last two layers are therefore the most topologically diverse, whereas the structure of the first two layers is constant (all trees have channels and functions). A very small fraction (< 1%) of trees that could not by parsed into our format was excluded from the dataset.
Table 6 shows various statistics about the topological characteristics of the recipes in the IFTTT dataset. The middle columns show percentage of trees that contain nonempty arguments and parameters in trigger (IF) and action (THEN) branches. Almost all recipes have none empty arguments and parameters (and thus depth 4, excluding the root), and a lower percentage—but still a majority—has arguments and parameters on the trigger side too. The last two columns show tree statistics pertaining to the complexity of trees after conversion to our format. The distribution of tree sizes is mostly concentrated between 4 and 30 nodes, with a slow-decaying tail of examples above this range (see Figure 8).
Regarding the content of the trees, the labels of the nodes in the first two levels (channels and functions) come from somewhat reduced vocabularies: 111 and 434 unique symbols for the trigger branch, respectively, and 157 and 85 for the action branch. The lower layers of the tree have a much more diverse vocabulary, with about 60K unique tokens in total. On the source side, the vocabulary over the sentence descriptions is large too, with about 30K unique tokens. The average sentence size is 6.07 tokens, with 80% of the sentences having at most 12 tokens.
C.3 MACHINE TRANSLATION
Starting from a preprocessed6 2% sub-selection of the English-French section of the WMT14 dataset, we further prune down the data by keeping only sentences of length between 5 and 20 words, and for which every word is within the 20K most frequent. The reason for this is to simplify the task by keeping only common words and avoiding out-of-vocabulary tokens. After this filtering, we are left with 53,607, 918 and 371 sentences for train, validation and test sets. After tokenizing, we obtain dependency parses for the target (English) sentences using the Stanford CoreNLP toolkit (Manning et al., 2014).
For the perturbation experiments, we randomly selected 50 sentences from among those in the test that could be easily restructured without significantly altering their meaning. The type of alterations we perform are: subordinate clause swapping, alternative construction substitution, passive/active voice change. In doing this, we try to keep the number of added/deleted words to a minimum, to minimize vocabulary-induced likelihood variations. When inserting new words, be verify that they are contained in the original vocabulary of 20K words. In Table 7 we show a few examples of the source, original target and perturbed target sentences.
6http://www-lium.univ-lemans.fr/ schwenk/cslm joint paper/
D ADDITIONAL EXAMPLE GENERATED TREES

Authors' response well answered my questions. Thanks.
Evaluation not changed.
###
This paper proposes a neural model for generating tree structure output from scratch. The model does 1) separate the recurrence between depths and siblings; 2) separate the topology and label generation, and outperforms previous methods on a benchmark IFTTT dataset. Compared to previous tree-decoding methods, the model avoids manually annotating subtrees with special tokens, and thus is a very good alternative to such problems. The paper does solid experiments on one synthetic dataset, and outperforms alternative methods on one real-world IFTTT dataset.
There are couple of interesting results in the paper that I believe is worth further investigation. Firstly, on the synthetic dataset, the precision drops rapidly with the number of nodes. Is it because that the vector representation of the sequential encoder fails to provide sufficient information of long sequences, such that the tree decoder can not do a good job? Or is it because that such tree decoder is not tolerant to the long sequence input, i.e., large tree structure? I believe that it is important to understand this before a better model can be developed. For example, if it is the fault of encoder, maybe an attention layer can be added, as in a seq-to-seq model, to preserve more information of the input sequence.
Moreover, besides only showing how the precision changes with the number of nodes in the tree, it might be interesting to investigate how it goes with 1) number of depths; 2) number of widths; 3) symmetricity; etc. Moreover, as greedy search is used in decoding, it might be interesting to see how it helps, if it does, to use beam-search in tree decoding.
On the IFTTT dataset, listing more statistics about this dataset might be helpful for better understanding the difficulty of this task. How deep are the trees? How large are the vocabularies on both language and program sides?
The paper is well written, except for minor typo as mentioned in my pre-review questions.
In general, I believe this is a solid paper, and more can be explored in this direction. So I tend to accept it.


INTROSPECTION:ACCELERATING NEURAL NETWORK TRAINING BY LEARNING WEIGHT EVOLUTION
Neural Networks are function approximators that have achieved state-of-the-art accuracy in numerous machine learning tasks. In spite of their great success in terms of accuracy, their large training time makes it difficult to use them for various tasks. In this paper, we explore the idea of learning weight evolution pattern from a simple network for accelerating training of novel neural networks. We use a neural network to learn the training pattern from MNIST classification and utilize it to accelerate training of neural networks used for CIFAR-10 and ImageNet classification. Our method has a low memory footprint and is computationally efficient. This method can also be used with other optimizers to give faster convergence. The results indicate a general trend in the weight evolution during training of neural networks.
Neural Networks are function approximators that have achieved state-of-the-art accuracy in numerous machine learning tasks. In spite of their great success in terms of accuracy, their large training time makes it difficult to use them for various tasks. In this paper, we explore the idea of learning weight evolution pattern from a simple network for accelerating training of novel neural networks.
We use a neural network to learn the training pattern from MNIST classification and utilize it to accelerate training of neural networks used for CIFAR-10 and ImageNet classification. Our method has a low memory footprint and is computationally efficient. This method can also be used with other optimizers to give faster convergence. The results indicate a general trend in the weight evolution during training of neural networks.
1 INTRODUCTION
Deep neural networks have been very successful in modeling high-level abstractions in data. However, training a deep neural network for any AI task is a time-consuming process. This is because a large number of parameters need to be learnt using training examples. Most of the deeper networks can take days to get trained even on GPU thus making it a major bottleneck in the large-scale application of deep networks. Reduction of training time through an efficient optimizer is essential for fast design and testing of deep neural nets.
In the context of neural networks, an optimization algorithm iteratively updates the parameters (weights) of a network based on a batch of training examples, to minimize an objective function. The most widely used optimization algorithm is Stochastic Gradient Descent. Even with the advent of newer and faster optimization algorithms like Adagrad, Adadelta, RMSProp and Adam there is still a need for achieving faster convergence.
In this work we apply neural network to predict weights of other in-training neural networks to accelerate their convergence. Our method has a very low memory footprint and is computationally efficient. Another aspect of this method is that we can update the weights of all the layers in parallel.
∗This work was done as part of an internship at Adobe Systems, Noida
2 RELATED WORK
Several extensions of Stochastic Gradient Descent have been proposed for faster training of neural networks. Some of them are Momentum (Rumelhart et al., 1986), AdaGrad (Duchy et al., 2011), AdaDelta (Zeiler, 2012), RMSProp (Hinton et al., 2012) and Adam (Kingma & Ba, 2014). All of them reduce the convergence time by suitably altering the learning rate during training. Our method can be used along with any of the above-mentioned methods to further improve convergence time.
In the above approaches, the weight update is always a product of the gradient and the modified/unmodified learning rate. More recent approaches (Andrychowicz et al., 2016) have tried to learn the function that takes as input the gradient and outputs the appropriate weight update. This exhibited a faster convergence compared to a simpler multiplication operation between the learning rate and gradient. Our approach is different from this, because our forecasting Network does not use the current gradient for weight update, but rather uses the weight history to predict its future value many time steps ahead where network would exhibit better convergence. Our approach generalizes better between different architectures and datasets without additional retraining. Further our approach has far lesser memory footprint as compared to (Andrychowicz et al., 2016). Also our approach need not be involved at every weight update and hence can be invoked asynchronously which makes it computationally efficient.
Another recent approach, called Q-gradient descent (Fu et al., 2016), uses a reinforcement learning framework to tune the hyperparameters of the optimization algorithm as the training progresses. The Deep-Q Network used for tuning the hyperparameters itself needs to be trained with data from any specific network N to be able to optimize the training of N . Our approach is different because we use a pre-trained forecasting Network that can optimize any network N without training itself by data from N .
Finally the recent approach by (Jaderberg et al., 2016) to predict synthetic gradients is similar to our work, in the sense that the weights are updates independently, but it still relies on an estimation of the gradient, while our update method does not.
Our method is distinct from all the above approaches because it uses information obtained from the training process of existing neural nets to accelerate the training of novel neural nets.
3 PATTERNS IN WEIGHT EVOLUTION
The evolution of weights of neural networks being trained on different classification tasks such as on MNIST and CIFAR-10 datasets and over different network architectures (weights from different layers of fully connected as well as convolutional architectures) as well as different optimization rules were analyzed. It was observed that the evolution followed a general trend independent of the task the model was performing or the layer to which the parameters belonged to. A major proportion of the weights did not undergo any significant change. Two metrics were used to quantify weight changes:
• Difference between the final and initial values of a weight scalar: This is a measure of how much a weight scalar has deviated from its initial value after training.In figure 4 we show the frequency histogram plot of the weight changes in a convolutional network trained for MNIST image classification task, which indicates that most of the weight values do not undergo a significant change in magnitude. Similar plots for a fully connected network trained on MNIST dataset ( figure 6 ) and a convolutional network trained on CIFAR-10 dataset (figure 8 ) present similar observations.
• Square root of 2nd moment of the values a weight scalar takes during training: Through this measure we wish to quantify the oscillation of weight values. This moment has been taken about the initial value of the weight. In figure 5, we show the frequency histogram plot of the second moment of weight changes in a convolutional network trained for the MNIST digit classification task, which indicates that most of the weight values do not undergo a significant oscillations in value during the training. Similar plots for a fully
connected network trained on MNIST (figure 7 ) and a convolutional network trained on CIFAR-10 ( figure 9) dataset present similar observations.
A very small subset of the all the weights undergo massive changes compared to the rest.
The few that did change significantly were observed to be following a predictable trend, where they would keep on increasing or decreasing with the progress of training in a predictable fashion. In figures 1, 2 and 3 we show the evolution history of a few weights randomly sampled from the weight change histogram bins of figures 4,6 and 8 respectively, which illustrates our observation.
0 20000 40000 60000 80000 100000 Training steps
−0.8
−0.6
−0.4
−0.2
0.0
0.2
0.4
0.6
0.8
Di ffe
re nc e of w ei gh
t al ue
fr om
in iti al iz ed
al ue
De iation of weight alue from initialization with training fully connected network on MNIST
Figure 2: Deviation of weight values from initialized values as a fully-connected network gets trained on MNIST dataset using Adam optimizer..
0 10000 20000 30000 40000 50000 Training s eps
−0.20
−0.15
−0.10
−0.05
0.00
0.05
0.10
Di ffe
re nc
e of
w ei
gh v
al ue
fr om
in i i
al iz
ed v
al ue
Devia ion of weigh values from ini ialized values when raining a convolu ional ne work on CIFAR-10
Figure 3: Deviation of weight values from initialized values as CNN gets trained on CIFAR-10 dataset using SGD optimizer.
3.1 WEIGHT PREDICTION
We collect the weight evolution trends of a network that is being trained and use the collected data to train a neural network I to forecast the future values of each weight based on its values in the previous time steps. The trained network I is then used to predict the weight values of an unseen network N during its training which move N to a state that enables a faster convergence. The time taken for the forecast is significantly smaller compared to the time a standard optimizer (e.g. SGD) would have taken to achieve the same accuracy. This leads to a reduction in the total training
time. The predictor I that is used for forecasting weights is a comparatively smaller neural network, whose inference time is negligible compared to the training time of the network that needs to be trained(N). We call this predictor I Introspection network because it looks at the weight evolution during training.
The forecasting network I is a simple 1-layered feedforward neuralnet. The input layer consists of four neurons that take four samples from the training history of a weight. The hidden layer consists of 40 neurons, fully connected to the input layer, with ReLU activation. The output layer is a single neuron that outputs the predicted future value of the weight. In our experiments four was minimum numbers of samples for which the training of Introspection Network I converged.
The figure 10 below shows a comparison of the weight evolution for a single scalar weight value with and without using the introspection network I . The vertical green bars indicate the points at which the introspection network was used to predict the future values. Post prediction, the network continues to get trained normally by SGD, until the introspection network I is used once again to jump to a new weight value.
4 EXPERIMENTS
4.1 TRAINING OF INTROSPECTION NETWORK
The introspection network I is trained on the training history of the weights of a network N0 which was trained on MNIST dataset.The network N0 consisted of 3 convolutional layers and two fully connected layers, with ReLU activation and deploying Adam optimiser. Max pooling(2X2 pool size and a 2X2 stride) was applied after the conv layers along with dropout applied after the first fc layer. The shapes of the conv layer filters were [5, 5, 1, 8] , [5, 5, 8, 16] and [5, 5, 16, 32] respectively whereas of the fc layer weight were [512, 1024] and [1024, 10] respectively.The network N0 was trained with a learning rate of 1e − 4 and batch size of 50. The training set of I is prepared as follows. A random training step t is selected for each weight of N0 selected as a training sample and the following 4 values are given as inputs for training I:
1. value of the weight at step t
2. value of the weight at step 7t/10
3. value of the weight at step 4t/10
4. at step 0 (i.e. the initialized value)
Since a large proportion of weights remain nearly constant throughout the training, a preprocessing step is done before getting the training data for I. The large number of weight histories collected are sorted in decreasing order on the basis of their variations in values from time step 0 to time step t. We choose 50% of the training data from the top 50th percentile of the sorted weights, 25% from the next 25th percentile(between 50 to 75th percentile of the sorted weights) and the remaining 25% from the rest (75th to 100th percentile). Approximately 0.8 million examples of weight history are used to train I . As the weight values are very small fractions they are further multiplied by 1000 before being input to the network I. The expected output of I , which is used for training I using backpropagation, is a single scalar the value of the same weight at step 2t. This is an empirical choice. For example, any step kt with k > 1 can be chosen instead of 2t. In our experiments with varying the value of k, we found that the value of k = 2.2 reached a slightly better validation accuracy than k = 2.0 on MNIST dataset (see figure 15 ) but, on the whole the value of k = 2.0 was a lot more consistent in its out-performance at various points in its history. All the results reported here are with respect to the I trained to predict weight values at 2t.
Adam optimizer was used for the training of the introspection network with a mini-batch size of 20.The training was carried out for 30k steps. The learning rate used was 5e-4 which decreased gradually after every 8k training steps. L1- error was used as the loss function for training . We experimented with both L2 error and percentage error but found that L1 error gave the best result over the validation set. The final training loss obtained was 3.1 and the validation loss of the final trained model was 3.4. These correspond to average L1 weight prediction error of 0.0031 and 0.0034 in the training and validation set respectively as the weight values are multiplied by 1000 before they are input to I .
4.2 USING PRE-TRAINED INTROSPECTION NETWORK TO TRAIN UNSEEN NETWORKS
The introspection network once trained can be then used to guide the training of other networks. We illustrate our method by using it to accelerate the training of several deep neural nets with varying architectures on 3 different datasets, namely MNIST, CIFAR-10 and ImageNet. We note that the same introspection network I , trained on the weight evolutions of the MNIST network N0 was used in all these different cases.
All the networks have been trained using either Stochastic Gradient Descent, or ADAM and the network I is used at a few intermediate steps to propel the network to a state with higher accuracy.We refer to the time step at which the introspection network I is applied to update all the weights as a ”jump point”.
The selection of the steps at which I is to be used is dependent on the distribution of the training step t used for training I . We show the effect of varying the timing of the initial jump and the time interval between jump points in section 4.2.2. It has been observed that I gives a better increase in accuracy when it is used in later training steps rather than in the earlier ones.
All the networks trained using I required comparatively less time to reach the same accuracy as normal SGD training. Also, when the same network was trained for the same time with and without updates by I , the former is observed to have better accuracy. These results show that there is a remarkable similarity in the weight evolution trajectories across network architectures,tasks and datasets.
4.2.1 MNIST
Four different neural networks were trained using I on MNIST dataset:
1. A convolutional neural network MNIST1 with 2 convolutional layer and 2 fully connected layers(dropout layer after 1st fc layer is also present)with ReLU acitvations for
classification task on MNIST image dataset.Max pooling(2X2 pool size and a 2X2 stride) was applied after every conv layer. The CNN layer weights were of shape [5, 5, 1, 8] and [5, 5, 32, 64] respectively and the fc layer were of sizes [3136, 1024] and [1024, 10].The weights were initialised from a truncated normal distribution with a mean of 0 and std of 0.01. The network was trained using SGD with a learning rate of 1e−2 and batch size of 50. It takes approximately 20,000 steps for convergence via SGD optimiser. For MNIST1, I was used to update all weights at training step 3000, 4000, and 5000.
2. A convolutional network MNIST2 with 2 convolutional layer and 2 fully connected layers with ReLU acitvations. Max pooling(2X2 pool size and a 2X2 stride) was applied after every conv layer. The two fc layer were of sizes [800, 500] and [500, 10] whereas the two conv layers were of shape [5, 5, 1, 20] and [5, 5, 20, 50] respectively. The weight initialisations were done via xavier intialisation. The initial learning rate was 0.01 which was decayed via the inv policy with gamma and power being 1e − 4 and 0.75 respectively. Batch size of 64 was used for the training.It takes approximately 10,000 steps for convergence . The network I was used to update weights at training step 2500 and 3000.
3. A fully connected network MNIST3 with 2 hidden layers each consisting of 256 hidden units and having ReLU acitvations. The network was trained using SGD with a learning rate of 5e − 3 and a batch size of 100. The initial weights were drawn out from a normal distribution having mean 0 and std as 1.0. For this network the weight updations were carried out at steps 6000, 8000 and 10000.
4. A RNN MNIST4 used to classify MNIST having a LSTM cell of hidden size of 128 followed by a fc layer of shape [128, 10] for classification. The RNN was trained on Adam optimizer with a learning rate of 5e− 4 and a batch size of 128. The weight updations for this network were done at steps 2000,3000 and 4000. Since the LSTM cell uses sigmoid and tanh activations, the RNN MNIST4 allows us to explore if the introspection network, trained on ReLU can generalize to networks using different activation functions.
A comparison of the validation accuracy with and without updates by I is shown in figures 11, 12 ,13 and 14. The green lines indicate the steps at which the introspection network I is used. For the MNIST1 network with the application of the introspection network I at three points, we found that it took 251 seconds and 20000 SGD steps to reach a validation accuracy of 98.22%. In the same number of SGD steps, normal training was able to reach a validation accuracy of only 97.22%. In the same amount of time (251 seconds), normal training only reached 97.92%. Hence the gain in accuracy with the application of introspection network translates to real gains in training times.
For the MNIST2 network, the figure 12 shows that to reach an accuracy of 99.11%, the number of iterations required by normal SGD was 6000, whereas with the application of the introspection network I , the number of iterations needed was only 3500, which represents a significant savings in time and computational effort.
training steps
The initial drop in accuracy seen after a jump in MNIST2 figure 12 can be attributed to the fact that each weight scalar is predicted independently, and the interrelationship between the weight scalars in a layer or across different layers is not taken into consideration. This interrelationship is soon reestablished after few SGD steps. This phenomenon is noticed in the CIFAR and ImageNet cases too.
training steps
training steps
For MNIST3 after 15000 steps of training,the max accuracy achieved by normal training of network via Adam optimizer was 95.71% whereas with introspection network applied the max accuracy was 96.89%. To reach the max accuracy reached by normal training , the modified network(weights updated by I) took only 8300 steps.
For MNIST4 after 7000 steps of training, the max accuracy achieved by normal training of network was 98.65% achieved after 6500 steps whereas after modification by I it was 98.85% achieved after 5300 steps. The modified network(weights updated by I) reached the max accuracy achieved by normal network after only 4200 steps. It is notable that the introspection network I trained on weight evolutions with ReLU activations was able to help accelerate the convergence of an RNN network which uses sigmoid and tanh activations.
4.2.2 CIFAR-10
We applied our introspection network I on a CNN CIFAR1 for classifying images in the CIFAR10 (Krizhevsky, 2009) dataset. It has 2 convolutional layers, 2 fully connected layer and a final softmax layer with ReLU activation function. Max pooling (3X3 pool size and a 2X2 stride) and batch normalization has been applied after each convolutional layer. The two conv layer filter weights were of shape [5, 5, 3, 64] and [5, 5, 64, 64] respectively whereas the two fc layers and final softmax layer were of shape [2304, 384],[384, 192] and [192, 10] respectively. The weights were initialized from a zero mean normal distribution with std of 1e − 4 for conv layers,0.04 for the two fc layers and 1/192.0 for the final layer. The initial learning rate used is 0.1 which is decayed by a factor of 0.1 after every 350 epochs. Batch size of 128 was used for training of the model which was trained via the SGD optimizer. It takes approximately 40,000 steps for convergence. The experiments on
CIFAR1 were done to investigate two issues. The first was to investigate if the introspection network trained on MNIST weight evolutions is able to generalize to a different network and different dataset. The second was to investigate the effect of varying the timing of the initial jump, the interval between successive jumps and the number of jumps. To investigate these issues, four separate training instances were performed with 4 different set of jump points:
1. Set1 : Weight updates were carried out at training steps 12000 and 17000. 2. Set2 : Weight updates at steps 15000 and 18000 . 3. Set3 : Weight updates at steps 12000 , 15000 and 19000 . 4. Set4 : Weight updates at steps 14000 , 17000 and 20000 .
We observed that for the CIFAR1 network that in order to reach a validation accuracy of 85.7%, we need 40,000 iterations with normal SGD without any intervention with the introspection network I . In all the four sets where the introspection network was used, the target accuracy of 85.7% was reached in approximately 28,000 steps. This shows that the introspection network is able to successfully generalize to a new dataset and new architecture and show significant gains in training time.
On CIFAR1, the time taken by I for prediction is negligible compared to the time required for SGD. So the training times in the above cases on CIFAR1 can be assumed to be proportional to the number of SGD steps required.
A comparison of the validation accuracy with and without updates by I at the four different sets of jump points are shown in figures 16, 17, 18 and 19. The results show that the while choice of jump points have some effect on the final result, the effects are not very huge. In general, we notice that better accuracy is reached when the jumps take place in later training steps.
4.2.3 IMAGENET
To investigate the practical feasibility and generalization ability of our introspection network, we applied it in training AlexNet(Krizhevsky et al., 2012) (AlexNet1) on the ImageNet (Russakovsky
et al., 2015) dataset. It has 5 conv layers and 3 fully connected layers . Max pooling and local response normalization have been used after the two starting conv layers and the pooling layer is there after the fifth conv layer as well. We use SGD with momentum of 0.9 to train this network, starting from a learning rate of 0.01. The learning rate was decreased by one tenth every 100, 000 iterations. The mini-batch size was 128. It takes approximately 300,000 steps for convergence. The weight updates were carried out at training steps 120, 000 , 130, 000 , 144, 000 and 160, 000 .
We find that in order to achieve a top-5 accuracy of 72%, the number of iterations required in the normal case was 196,000. When the introspection network was used, number of iterations required to reach the same accuracy was 179,000. Again the time taken by I for prediction is negligible compared to the time required for SGD. A comparison of the validation accuracy with and without updates by I is shown in figure 20. The green lines indicate the steps at which the introspection network I is used. The corresponding plot of loss function against training steps has been shown in figure 21.
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2
Training steps ×10 5
0.6
0.62
0.64
0.66
0.68
0.7
0.72
0.74
A c c u ra
c y
Plot of accuracy vs training steps for imageNet
normal training Introspection network applied
Figure 20: Validation accuracy plot for AlexNet1 on ImageNet
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8
x 10 5
1.6
1.8
2
2.2
2.4
2.6
2.8
3
3.2
Training steps
T ra
in in
g l o
s s
normal training Introspection network applied
Figure 21: Plot of loss function vs training steps for AlexNet1 on ImageNet
The results on Alexnet1 show that our approach has a small memory footprint and computationally efficient to be able to scale to training practical large scale networks.
4.3 COMPARISON WITH BASELINE TECHNIQUES
In this section we provide a comparison with other optimizers and simple heuristics which can be used to update the weights at different training steps instead of updations by introspection network.
4.4 COMPARISON WITH ADAM OPTIMIZER
We applied the introspection network on MNIST1 and MNIST3 networks being trained with Adam optimizer with learning rates of 1e − 4 and 1e − 3. The results in figure 22 and figure 23 show that while Adam outperforms normal SGD and SGD with introspection, we were able to successfully apply the introspection network on Adam optimizer and accelerate it.
For MNIST1 the max accuracy achieved by Adam with introspection was 99.34%, by normal Adam was 99.3%, by SGD with introspection was 99.21% and by normal SGD was 99.08% . With introspection applied on Adam the model reaches the max accuracy as achieved by normal Adam after only 7200 steps whereas the normal training required 10000 steps.
For MNIST3 the max accuracy achieved by Adam with introspection was 96.9%, by normal Adam was 95.7%, by SGD with introspection was 94.47% and by normal SGD was 93.39% . With introspection applied on Adam the model reaches the max accuracy as achieved by normal Adam after only 8800 steps whereas the normal training required 15000 steps.
training steps
training steps
4.4.1 FITTING QUADRATIC CURVE
A separate quadratic curve was fit to each of the weight values of the model on the basis of the 4 past weight values chosen from history.The weight values chosen from history were at the same steps as they were for updations by I . The new updated weight would be the value of the quadratic curve at some future time step.For MNIST1 , experiments were performed by updating the weights to the value predicted by the quadratic function at a future timestep which was one of 1.25,1.3 or 1.4 times the current time step. For other higher jump ratios the updates would cause the model to diverge, and lower jump ratios did not show much improvement in performance. The plot showing the comparison in validation accuracy have been shown below in figure 24.
training steps
The max accuracy achieved with introspection applied was 99.21% whereas with quadratic fit it was 99.19%. We note that even though the best performing quadratic fit eventually almost reaches the same max accuracy than that achieved with introspection network, it required considerable experimentation to find the right jump ratio.A unique observation for the quadratic fit baseline was that it would take the accuracy down dramatically, upto 9.8%, from which the training often never recovers. Sometimes,the optimizers (SGD or Adam) would recover the accuracy, as seen in figure 24. Moreover, the quadratic fit baseline was not able to generalize to other datasets and tasks. The best performing jump ratio of 1.25 was not able to outperform Introspection on the CIFAR-10 dataset, as seen in figure 25.
In the CIFAR-10 case, The maximum accuracy achieved via updations by introspection was 85.6 which was achieved after 25500 steps, whereas with updations by quadratic fit, the max accuracy of 85.45 was achieved after 27200 steps.
For the normal training via SGD without any updations after 30000 steps of training, the max accuracy of 85.29 was achieved after 26500 steps, whereas the same accuracy was achieved by introspection after only 21200 steps and after 27000 steps via updation by quadratic.
4.4.2 FITTING LINEAR CURVE
Instead of fitting a quadratic curve to each of the weights we tried fitting a linear curve. Experiments were performed on MNIST1 for jump ratios of 1.1 and 1.075 as the higher ratios would cause the model to diverge after 2 or 3 jumps.The result has been shown below in figure 26.
training steps
training steps
As no significant improvement in performance was observed the experiment was not repeated over cifar.
4.5 LINEAR INTROSPECTION NETWORK
We removed the ReLU nonlinearity from the introspection network and used the same training procedure of the normal introspection network to predict the future values at 2t. We then used this linear network on the MNIST1 network. We found that it gave some advantage over normal SGD, but was not as good as the introspection network as shown in figure 27. Hence we did not explore this baseline for other datasets and networks.
4.5.1 ADDING NOISE
The weight values were updated by adding small gaussian random zero mean noise values . The experiment was performed over MNIST1 for two different std. value, the results of which have been shown below in figure 28.
Since no significant improvement was observed for the weight updations via noise for MNIST, the experiment was not performed over cifar-10.
5 LIMITATIONS AND OPEN QUESTIONS
Some of the open questions to be investigated relate to determination of the optimal jump points and investigations regarding the generalization capacity of the introspection network to speed up training
in RNNs and non-image tasks. Also, we noticed that applying the jumps in very early training steps while training AlexNet1 tended to degrade the final outcomes. This may be due to the fact that our introspection network is extremely simple and has been trained only on weight evolution data from MNIST. A combination of a more powerful network and training data derived from a diverse set may ameliorate this problem.
6 CONCLUSION
We introduced a method to accelerate neural network training. For this purpose, we used a neural network I that learns a general trend in weight evolution of all neural networks. After learning the trend from one neural network training, I is used to update weights of many deep neural nets on 3 different tasks - MNIST, CIFAR-10, and ImageNet, with varying network architectures, activations, optimizers, and normalizing strategies(batch norm,lrn). Using the introspection network I led to faster convergence compared to existing methods in all the cases. Our method has a small memory footprint, is computationally efficient and is usable in practical settings. Our method is different from other existing methods in the aspect that it utilizes the knowledge obtained from weights of one neural network training to accelerate the training of several unseen networks on new tasks. The results reported here indicates the existence of a general underlying pattern in the weight evolution of any neural network.
A APPENDIX
In this section, we report some initial results of applying the introspection network I (trained on the weight evolution of MNIST network N0) to accelerate the training of inception v1 network (Szegedy et al., 2014). We trained the inception v1 network on imagenet dataset with a mini-batchsize of 128 and a RMS optimizer(decay 0.9, momentum 0.9, epsilon 1.0) starting from a learning rate of 0.01 with a decay of 0.94 after every 2 epochs. The network training is still in progress, and we will eventually report on the final outcome. However we thought it would be valuable to share the preliminary results all the same.
We found that applying introspection network seems to be reducing the training time quite significantly. In Figures 29 and 30, we see that applying the introspection network leads to a gain of at least 730,000 steps.After training for around 1.5 million steps, the maximum accuracy achieved by normal training was 68.40%, whereas with introspection applied after every 300k steps the max accuracy achieved was 69.06%.The network achieved the max accuracy of 68.40% after only 852k steps. With introspection applied at steps 200k, 400k and 600k the max accuracy achieved was 68.69% and it reached the max accuracy achieved by the normal training of model after only 944k steps.
However, we also observed that choosing the jump points early in the training does not lead to eventual gains, even though a significant jump in accuracy is observed initially. Figure 31 shows the flattening of the test accuracy after a set of early jumps. It remains to be seen if further interventions later in the training can help maintain the initial accelerated convergence.

EDIT: Updated score. See additional comment.
I quite like the main idea of the paper, which is based on the observation in Sec. 3.0 - that the authors find many predictable patterns in the independent evolution of weights during neural network training. It is very encouraging that a simple neural network can be used to speed up training by directly predicting weights.
However the technical quality of the current paper leaves much to be desired, and I encourage the authors to do more rigorous analysis of the approach. Here are some concrete suggestions:
- The findings in Section 3.0 which motivate the approach, should be clearly presented in the paper. Presently they are stated as anecdotes.
- A central issue with the paper is that the training of the Introspection network I is completely glossed over. How well did the training work, in terms of training, validation/test losses? How well does it need to work in order to be useful for speeding up training? These are important questions for anyone interested in this approach.
- An additional important issue is that of baselines. Would a simple linear/quadratic model also work instead of a neural network? What about a simple heuristic rule to increase/decrease weights? I think it's important to compare to such baselines to understand the complexity of the weight evolution learned by the neural network.
- I do not think that default tensorflow example hyperparameters should be used, as mentioned by authors on OpenReview. There is no scientific basis for using them. Instead, first hyperparameters which produce good results in a reasonable time should be selected as the baseline, and then added the benefit of the introspection network to speed up training (and reaching a similar result) should be shown.
- The authors state in the discussion on OpenReview that they also tried RNNs as the introspection network but it didn't work with small state size. What does "didn't work" mean in this context? Did it underfit? I find it hard to imagine that a large state size would be required for this task. Even if it is, that doesn't rule out evaluation due to memory issues because the RNN can be run on the weights in 'mini-batch' mode. In general, I think other baselines are more important than RNN.
- A question about jump points:
The I is trained on SGD trajectories. While using I to speed up training at several jump points, if the input weights cross previous jump points, then I gets input data from a weight evolution which is not from SGD (it has been altered by I). This seems problematic but doesn't seem to affect your experiments. I feel that this again highlights the importance of the baselines. Perhaps I is doing something extremely simple that is not affected by this issue.
Since the main idea is very interesting, I will be happy to update my score if the above concerns are addressed.