NEURO-SYMBOLIC PROGRAM SYNTHESIS
Recent years have seen the proposal of a number of neural architectures for the problem of Program Induction. Given a set of input-output examples, these architectures are able to learn mappings that generalize to new test inputs. While achieving impressive results, these approaches have a number of important limitations: (a) they are computationally expensive and hard to train, (b) a model has to be trained for each task (program) separately, and (c) it is hard to interpret or verify the correctness of the learnt mapping (as it is defined by a neural network). In this paper, we propose a novel technique, Neuro-Symbolic Program Synthesis, to overcome the above-mentioned problems. Once trained, our approach can automatically construct computer programs in a domain-specific language that are consistent with a set of input-output examples provided at test time. Our method is based on two novel neural modules. The first module, called the cross correlation I/O network, given a set of input-output examples, produces a continuous representation of the set of I/O examples. The second module, the RecursiveReverse-Recursive Neural Network (R3NN), given the continuous representation of the examples, synthesizes a program by incrementally expanding partial programs. We demonstrate the effectiveness of our approach by applying it to the rich and complex domain of regular expression based string transformations. Experiments show that the R3NN model is not only able to construct programs from new input-output examples, but it is also able to construct new programs for tasks that it had never observed before during training.
1 INTRODUCTION
The act of programming, i.e., developing a procedure to accomplish a task, is a remarkable demonstration of the reasoning abilities of the human mind. Expectedly, Program Induction is considered as one of the fundamental problems in Machine Learning and Artificial Intelligence. Recent progress on deep learning has led to the proposal of a number of promising neural architectures for this problem. Many of these models are inspired from computation modules (CPU, RAM, GPU) (Graves et al., 2014; Kurach et al., 2015; Reed & de Freitas, 2015; Neelakantan et al., 2015) or common data structures used in many algorithms (stack) (Joulin & Mikolov, 2015). A common thread in this line of work is to specify the atomic operations of the network in some differentiable form, allowing efficient end-to-end training of a neural controller, or to use reinforcement learning to make hard choices about which operation to perform. While these results are impressive, these approaches have a number of important limitations: (a) they are computationally expensive and hard to train, (b) a model has to be trained for each task (program) separately, and (c) it is hard to interpret or verify the correctness of the learnt mapping (as it is defined by a neural network). While some recently proposed methods (Kurach et al., 2015; Gaunt et al., 2016; Riedel et al., 2016; Bunel et al., 2016) do learn interpretable programs, they still need to learn a separate neural network model for each individual task.
Motivated by the need for model interpretability and scalability to multiple tasks, we address the problem of Program Synthesis. Program Synthesis, the problem of automatically constructing programs that are consistent with a given specification, has long been a subject of research in Computer Science (Biermann, 1978; Summers, 1977). This interest has been reinvigorated in recent years on
the back of the development of methods for learning programs in various domains, ranging from low-level bit manipulation code (Solar-Lezama et al., 2005) to data structure manipulations (Singh & Solar-Lezama, 2011) and regular expression based string transformations (Gulwani, 2011).
Most of the recently proposed methods for program synthesis operate by searching the space of programs in a Domain-Specific Language (DSL) instead of arbitrary Turing-complete languages. This hypothesis space of possible programs is huge (potentially infinite) and searching over it is a challenging problem. Several search techniques including enumerative (Udupa et al., 2013), stochastic (Schkufza et al., 2013), constraint-based (Solar-Lezama, 2008), and version-space algebra based algorithms (Gulwani et al., 2012) have been developed to search over the space of programs in the DSL, which support different kinds of specifications (examples, partial programs, natural language etc.) and domains. These techniques not only require significant engineering and research effort to develop carefully-designed heuristics for efficient search, but also have limited applicability and can only synthesize programs of limited sizes and types.
In this paper, we present a novel technique called Neuro-Symbolic Program Synthesis (NSPS) that learns to generate a program incrementally without the need for an explicit search. Once trained, NSPS can automatically construct computer programs that are consistent with any set of input-output examples provided at test time. Our method is based on two novel module neural architectures. The first module, called the cross correlation I/O network, produces a continuous representation of any given set of input-output examples. The second module, the Recursive-Reverse-Recursive Neural Network (R3NN), given the continuous representation of the input-output examples, synthesizes a program by incrementally expanding partial programs. R3NN employs a tree-based neural architecture that sequentially constructs a parse tree by selecting which non-terminal symbol to expand using rules from a context-free grammar (i.e., the DSL).
We demonstrate the efficacy of our method by applying it to the rich and complex domain of regularexpression-based syntactic string transformations, using a DSL based on the one used by FlashFill (Gulwani, 2011; Gulwani et al., 2012), a Programming-By-Example (PBE) system in Microsoft Excel 2013. Given a few input-output examples of strings, the task is to synthesize a program built on regular expressions to perform the desired string transformation. An example task that can be expressed in this DSL is shown in Figure 1, which also shows the DSL.
Our evaluation shows that NSPS is not only able to construct programs for known tasks from new input-output examples, but it is also able to construct completely new programs that it had not observed during training. Specifically, the proposed system is able to synthesize string transformation programs for 63% of tasks that it had not observed at training time, and for 94% of tasks when 100 program samples are taken from the model. Moreover, our system is able to learn 38% of 238 real-world FlashFill benchmarks.
To summarize, the key contributions of our work are:
• A novel Neuro-Symbolic program synthesis technique to encode neural search over the space of programs defined using a Domain-Specific Language (DSL).
• The R3NN model that encodes and expands partial programs in the DSL, where each node has a global representation of the program tree.
• A novel cross-correlation based neural architecture for learning continuous representation of sets of input-output examples.
• Evaluation of the NSPS approach on the complex domain of regular expression based string transformations.
2 PROBLEM DEFINITION
In this section, we formally define the DSL-based program synthesis problem that we consider in this paper. Given a DSL L, we want to automatically construct a synthesis algorithm A such that given a set of input-output example, {(i1, o1), · · · , (in, on)}, A returns a program P ∈ L that conforms to the input-output examples, i.e.,
∀j : 1 ≤ j ≤ n P (ij) = oj . (1)
The syntax and semantics of the DSL for string transformations is shown in Figure 1(b) and Figure 8 respectively. The DSL corresponds to a large subset of FlashFill DSL (except conditionals), and allows for a richer class of substring operations than FlashFill. A DSL program takes as input a string v and returns an output string o. The top-level string expression e is a concatenation of a finite list of substring expressions f1, · · · , fn. A substring expression f can either be a constant string s or a substring expression, which is defined using two position logics pl (left) and pr (right). A position logic corresponds to a symbolic expression that evaluates to an index in the string. A position logic p can either be a constant position k or a token match expression (r, k,Dir), which denotes the Start or End of the kth match of token r in input string v. A regex token can either be a constant string s or one of 8 regular expression tokens: p (ProperCase), C (CAPS), l (lowercase), d (Digits), α (Alphabets), αn (Alphanumeric), ∧ (StartOfString), and $ (EndOfString). The semantics of the DSL programs is described in the appendix.
A DSL program for the name transformation task shown in Figure 1(a) that is consistent with the examples is: Concat(f1,ConstStr(“, ”), f2,ConstStr(“.”)), where f1 ≡ SubStr(v, (“ ”,−1,End),ConstPos(−1)) and f2 ≡ SubStr(v,ConstPos(0),ConstPos(1)). The program concatenates the following 4 strings: i) substring between the end of last whitespace and end of string, ii) constant string “, ”, iii) first character of input string, and iv) constant string “.”.
3 OVERVIEW OF OUR APPROACH
We now present an overview of our approach. Given a DSL L, we learn a generative model of programs in the DSL L that is conditioned on input-output examples to efficiently search for consistent programs. The workflow of our system is shown in Figure 2, which is trained end-to-end using a large training set of programs in the DSL together with their corresponding input-output examples. To generate a large training set, we uniformly sample programs from the DSL and then use a rule-based strategy to compute well-formed input strings. Given a program P (sampled from the DSL), the rule-based strategy generates input strings for the program P ensuring that the preconditions of P are met (i.e. P doesn’t throw an exception on the input strings). It collects the pre-conditions of all Substring expressions present in the sampled program P and then generates inputs conforming to them. For example, let’s assume the sampled program is SubStr(v,(CAPS, 2, Start), (“ ”, 3, Start)), which extracts the substring between the start of 2nd capital letter and start of 3rd whitespace. The rule-based strategy would ensure that all the generated input strings consist of at least 2 capital letters and 3 whitespaces in addition to other randomly generated characters. The corresponding output strings are obtained by running the programs on the input strings.
A DSL can be considered as a context-free grammar with a start symbol S and a set of non-terminals with corresponding expansion rules. The (partial) grammar derivations or trees correspond to (partial) programs. A naı̈ve way to perform a search over the programs in a DSL is to start from the start symbol S and then randomly choose non-terminals to expand with randomly chosen expansion rules until reaching a derivation with only terminals. We, instead, learn a generative model over partial derivations in the DSL that assigns probabilities to different non-terminals in a partial derivation and corresponding expansions to guide the search for complete derivations.
Our generative model uses a Recursive-Reverse-Recursive Neural Network (R3NN) to encode partial trees (derivations) in L, where each node in the partial tree encodes global information about every other node in the tree. The model assigns a vector representation for every symbol and every expansion rule in the grammar. Given a partial tree, the model first assigns a vector representation to each leaf node, and then performs a recursive pass going up in the tree to assign a global tree representation to the root. It then performs a reverse-recursive pass starting from the root to assign a global tree representation to each node in the tree.
The generative process is conditioned on a set of input-output examples to learn a program that is consistent with this set of examples. We experiment with multiple input-output encoders including an LSTM encoder that concatenates the hidden vectors of two deep bidirectional LSTM networks for input and output strings in the examples, and a Cross Correlation encoder that computes the cross correlation between the LSTM tensor representations of input and output strings in the examples. This vector is then used as an additional input in the R3NN model to condition the generative model.
4 TREE-STRUCTURED GENERATION MODEL
We define a program t-steps into construction as a partial program tree (PPT) (see Figure 3 for a visual depiction). A PPT has two types of nodes: leaf (symbol) nodes and inner non-leaf (rule) nodes. A leaf node represents a symbol, whether non-terminal or terminal. An inner non-leaf node represents a particular production rule of the DSL, where the number of children of the non-leaf node is equivalent to the arity of the RHS of the rule it represents. A PPT is called a program tree (PT) whenever all the leaves of the tree are terminal symbols. Such a tree represents a completed program under the DSL and can be executed. We define an expansion as the valid application of a specific production rule (e → e op2 e) to a specific non-terminal leaf node within a PPT (leaf with symbol e). We refer to the specific production rule that an expansion is derived from as the expansion type. It can be seen that if there exist two leaf nodes (l1 and l2) with the same symbol then for every expansion specific to l1 there exists an expansion specific to l2 with the same type.
4.1 RECURSIVE-REVERSE-RECURSIVE NEURAL NETWORK
In order to define a generation model over PPTs, we need an efficient way of assigning probabilities to every valid expansion in the current PPT. A valid expansion has two components: first the production rule used, and second the position of the expanded leaf node relative to every other node in the tree. To account for the first component, a separate distributed representation for each production rule is maintained. The second component is handled using an architecture where the forward propagation resembles belief propagation on trees, allowing a notion of global tree state at every node within the tree. A given expansion probability is then calculated as being proportional to the inner product between the production rule representation and the global-tree representation of the leaf-level non-terminal node. We now describe the design of this architecture in more detail.
The R3NN has the following parameters for the grammar described by a DSL (see Figure 3):
1. For every symbol s ∈ S, an M−dimensional representation φ(s) ∈ RM . 2. For every production rule r ∈ R, an M−dimensional representation ω(r) ∈ RM .
3. For every production rule r ∈ R, a deep neural network fr which takes as input a vector x ∈ RQ·M , with Q being the number of symbols on the RHS of the production rule r, and outputs a vector y ∈ RM . Therefore, the production-rule network fr takes as input a concatenation of the distributed representations of each of its RHS symbols and produces a distributed representation for the LHS symbol.
4. For every production rule r ∈ R, an additional deep neural network gr which takes as input a vector x′ ∈ RM and outputs a vector y′ ∈ RQ·M . We can think of gr as a reverse production-rule network that takes as input a vector representation of the LHS and produces a concatenation of the distributed representations of each of the rule’s RHS symbols.
Let E be the set of all valid expansions in a PPT T , let L be the current leaf nodes of T and N be the current non-leaf (rule) nodes of T . Let S(l) be the symbol of leaf l ∈ L and R(n) represent the production rule of non-leaf node n ∈ N .
4.1.1 GLOBAL TREE INFORMATION AT THE LEAVES
To compute the probability distribution over the set E, the R3NN first computes a distributed representation for each leaf node that contains global tree information. To accomplish this, for every leaf node l ∈ L in the tree we retrieve its distributed representation φ(S(l)) . We now do a standard recursive bottom-to-top, RHS→LHS pass on the network, by going up the tree and applying fR(n) for every non-leaf node n ∈ N on its RHS node representations (see Figure 3(a)). These networks fR(n) produce a node representation which is input into the parent’s rule network and so on until we reach the root node.
Once at the root node, we effectively have a fixed-dimensionality global tree representation φ(root) for the start symbol. The problem is that this representation has lost any notion of tree position. To solve this problem R3NN now does what is effectively a reverse-recursive pass which starts at the root node with φ(root) as input and moves towards the leaf nodes (see Figure 3(b)).
More concretely, we start with the root node representation φ(root) and use that as input into the rule network gR(root) where R(root) is the production rule that is applied to the start symbol in T . This produces a representation φ′(c) for each RHS node c of R(root). If c is a non-leaf node, we iteratively apply this procedure to c, i.e., process φ′(c) using gR(c) to get representations φ′(cc) for every RHS node cc of R(c), etc. If c is a leaf node, we now have a leaf representation φ′(c) which has an information path to φ(root) and thus to every other leaf node in the tree. Once the reverse-recursive process is complete, we now have a distributed representation φ′(l) for every leaf node l which contains global tree information. While φ(l1) and φ(l2) could be equal for leaf nodes which have the same symbol type, φ′(l1) and φ′(l2) will not be equal even if they have the same symbol type because they are at different positions in the tree.
4.1.2 EXPANSION PROBABILITIES
Given the global leaf representations φ′(l), we can now straightforwardly acquire scores for each e ∈ E. For expansion e, let e.r be the expansion type (production rule r ∈ R that e applies) and let e.l be the leaf node l that e.r is applied to. ze = φ′(e.l) · ω(e.r) The score of an expansion is calculated using ze = φ′(e.l) · ω(e.r). The probability of expansion e is simply the exponentiated normalized sum over all scores: π(e) = e
ze∑ e′∈E e z e′ .
An additional improvement that was found to help was to add a bidirectional LSTM (BLSTM) to process the global leaf representations right before calculating the scores. To do this, we first order the global leaf representations sequentially from left-most leaf node to right-mode leaf node. We then treat each leaf node as a time step for a BLSTM to process. This provides a sort of skip connection between leaf nodes, which potentially reduces the path length that information needs to travel between leaf nodes in the tree. The BLSTM hidden states are then used in the score calculation rather than the leaves themselves.
The R3NN can be seen as an extension and combination of several previous tree-based models, which were mainly developed in the context of natural language processing (Le & Zuidema, 2014; Paulus et al., 2014; Irsoy & Cardie, 2013).
5 CONDITIONING WITH INPUT/OUTPUT EXAMPLES
Now that we have defined a generation process over tree-structured programs, we need a way of conditioning this generation process on a set of input/output examples. The set of input/output examples provide a nearly complete specification for the desired output program, and so a good encoding of the examples is crucial to the success of our program generator. For the most part, this example encoding needs to be domain-specific, since different DSLs have different inputs (some may operate over integers, some over strings, etc.). Therefore, in our case, we use an encoding adapted to the input-output strings that our DSL operates over. We also investigate different ways of conditioning program search on the learnt example input-output encodings.
5.1 ENCODING INPUT/OUTPUT EXAMPLES
There are two types of information that string manipulation programs need to extract from inputoutput examples: 1) constant strings, such as “@domain.com” or “.”, which appear in all output examples; 2) substring indices in input where the index might be further defined by a regular expression. These indices determine which parts of the input are also present in the output. To simplify the DSL, we assume that there is a fixed finite universe of possible constant strings that could appear in programs. Therefore we focus on extracting the second type of information, the substring indices.
In earlier hand-engineered systems such as FlashFill, this information was extracted from the inputoutput strings by running the Longest Common Substring algorithm, a dynamic programming algorithm that efficiently finds matching substrings in string pairs. To extract substrings, FlashFill runs LCS on every input-output string pair in the I/O set to get a set of substring candidates. It then takes the entire set of substring candidates and simply tries every possible regex and constant index that can be used at substring boundaries, exhaustively searching for the one which is the most “general”, where generality is specified by hand-engineered heuristics.
In contrast to these previous methods, instead of hand-designing a complicated algorithm to extract regex-based substrings, we develop neural network based architectures that are capable of learning to extract and produce continuous representations of the likely regular expressions given I/O examples.
5.1.1 BASELINE LSTM ENCODER
Our first I/O encoding network involves running two separate deep bidirectional LSTM networks for processing the input and the output string in each example pair. For each pair, it then concatenates the topmost hidden representation at every time step to produce a 4HT -dimensional feature vector per I/O pair, where T is the maximum string length for any input or output string, and H is the topmost LSTM hidden dimension.
We then concatenate the encoding vectors across all I/O pairs to get a vector representation of the entire I/O set. This encoding is conceptually straightforward and has very little prior knowledge about what operations are being performed over the strings, i.e., substring, constant, etc., which might make it difficult to discover substring indices, especially the ones based on regular expressions.
5.1.2 CROSS CORRELATION ENCODER
To help the model discover input substrings that are copied to the output, we designed an novel I/O example encoder to compute the cross correlation between each input and output example representation. We used the two output tensors of the LSTM encoder (discussed above) as inputs to this encoder. For each example pair, we first slide the output feature block over the input feature block and compute the dot product between the respective position representation. Then, we sum over all overlapping time steps. Features of all pairs are then concatenated to form a 2∗ (T −1)-dimensional vector encoding for all example pairs. There are 2 ∗ (T − 1) possible alignments in total between input and output feature blocks. An illustration of the cross-correlation encoder is shown in Figure 9. We also designed the following variants of this encoder.
Diffused Cross Correlation Encoder: This encoder is identical to the Cross Correlation encoder except that instead of summing over overlapping time steps after the element-wise dot product, we simply concatenate the vectors corresponding to all time steps, resulting in a final representation that contains 2 ∗ (T − 1) ∗ T features for each example pair. LSTM-Sum Cross Correlation Encoder: In this variant of the Cross Correlation encoder, instead of doing an element-wise dot product, we run a bidirectional LSTM over the concatenated feature blocks of each alignment. We represent each alignment by the LSTM hidden representation of the final time step leading to a total of 2 ∗H ∗ 2 ∗ (T − 1) features for each example pair. Augmented Diffused Cross Correlation Encoder: For this encoder, the output of each character position of the Diffused Cross Correlation encoder is combined with the character embedding at this position, then a basic LSTM encoder is run over the combined features to extract a 4∗H-dimensional vector for both the input and the output streams. The LSTM encoder output is then concatenated with the output of the Diffused Cross Correlation encoder forming a (4∗H+T ∗(T−1))-dimensional feature vector for each example pair.
5.2 CONDITIONING PROGRAM SEARCH ON EXAMPLE ENCODINGS
Once the I/O example encodings have been computed, we can use them to perform conditional generation of the program tree using the R3NN model. There are a number of ways in which the PPT generation model can be conditioned using the I/O example encodings depending on where the I/O example information is inserted in the R3NN model. We investigated three locations to inject example encodings:
1) Pre-conditioning: where example encodings are concatenated to the encoding of each tree leaf, and then passed to a conditioning network before the bottom-up recursive pass over the program tree. The conditioning network can be either a multi-layer feedforward network, or a bidirectional LSTM network running over tree leaves. Running an LSTM over tree leaves allows the model to learn more about the relative position of each leaf node in the tree.
2) Post-conditioning: After the reverse-recursive pass, example encodings are concatenated to the updated representation of each tree leaf and then fed to a conditioning network before computing the expansion scores.
3) Root-conditioning: After the recursive pass over the tree, the root encoding is concatenated to the example encodings and passed to a conditioning network. The updated root representation is then used to drive the reverse-recursive pass.
Empirically, pre-conditioning worked better than either root- or post- conditioning. In addition, conditioning at all 3 places simultaneously did not cause a significant improvement over just pre-conditioning. Therefore, for the experimental section, we report models which only use preconditioning.
6 EXPERIMENTS
In order to evaluate and compare variants of the previously described models, we generate a dataset randomly from the DSL. To do so, we first enumerate all possible programs under the DSL up to a specific number of instructions, which are then partitioned into training, validation and test sets. In order to have a tractable number of programs, we limited the maximum number of instructions for programs to be 13. Length 13 programs are important for this specific DSL because all larger programs can be written as compositions of sub-programs of length at most 13. The semantics of length 13 programs therefore constitute the “atoms” of this particular DSL.
In testing our model, there are two different categories of generalization. The first is input/output generalization, where we are given a new set of input/output examples as well as a program with a specific tree that we have seen during training. This represents the model’s capacity to be applied on new data. The second category is program generalization, where we are given both a previously unseen program tree in addition to unseen input/output examples. Therefore the model needs to have a sufficient enough understanding of the semantics of the DSL that it can construct novel combinations of operations. For all reported results, training sets correspond to the first type of generalization since we have seen the program tree but not the input/output pairs. Test sets represent the second type of generalization, as they are trees which have not been seen before on input/output pairs that have also not been seen before.
In this section, we compare several different variants of our model. We first evaluate the effect of each of the previously described input/output encoders. We then evaluate the R3NN model against a simple recurrent model called io2seq, which is basically an LSTM that takes as input the input/output conditioning vector and outputs a sequence of DSL symbols that represents a linearized program tree. Finally, we report the results of the best model on the length 13 training and testing sets, as well as on a set of 238 benchmark functions.
6.1 SETUP AND HYPERPARAMETERS SETTINGS
For training the R3NN, two hyperparameters that were crucial for stabilizing training were the use of hyperbolic tangent activation functions in both R3NN (other activations such as ReLU more consistently diverged during our initial experiments) and cross-correlation I/O encoders and the use of minibatches of length 8. Additionally, for all results, the program tree generation is conditioned on a set of 10 input/output string pairs. We used ADAM (Kingma & Ba, 2014) to optimize the networks with a learning rate of 0.001. Network weights used the default torch initializations.
Due to the difficulty of batching tree-based neural networks since each sample in a batch has a potentially different tree structure, we needed to do batching sequentially. Therefore for each minibatch of size N , we accumulated the gradients for each sample. After all N sample gradients were accumulated, we updated the parameters and reset the accumulated gradients. Due to this sequential processing, in order to train models in a reasonable time, we limited our batch sizes to between 8-12. Despite the computational inefficiency, batching was critical to successfully train an R3NN, as online learning often caused the network to diverge.
For each latent function and set of input/output examples that we test on, we report whether we had a success after sampling 100 functions from the model and testing all 100 to see if one of these functions is equivalent to the latent function. Here we consider two functions to be equivalent with respect to a specific input/output example set if the functions output the same strings when run on the inputs. Under this definition, two functions can have a different set of operations but still be equivalent with respect to a specific input-output set.
We restricted the maximum size of training programs to be 13 because of two computational considerations. As described earlier, one difficulty was in batching tree-based neural networks of different structure and the computational cost of batching increases with the increase in size of the program trees. The second issue is that valid I/O strings for programs often grow with the program length, in the sense that for programs of length 40 a minimal valid I/O string will typically be much longer than a minimal valid I/O string for length 20 programs. For example, for a program such as (Concat (ConstStr “longstring”) (Concat (ConstStr “longstring”) (Concat (ConstStr “longstring”) ...))), the valid output string would be “longstringlongstringlongstring...” which could be many
hundreds of characters long. Because of limited GPU memory, the I/O encoder models can quickly run out of memory.
6.2 EXAMPLE ENCODING
In this section, we evaluate the effect of several different input/output example encoders. To control for the effect of the tree model, all results here used an R3NN with fixed hyperparameters to generate the program tree. Table 1 shows the performance of several of these input/output example encoders. We can see that the summed cross-correlation encoder did not perform well, which can be due to the fact that the sum destroys positional information that might be useful for determining specific substring indices. The LSTM-sum and the augmented diffused cross-correlation models did the best. Surprisingly, the LSTM encoder was capable of finding nearly 88% of all programs without having any prior knowledge explicitly built into the architecture. We use 100 samples for evaluating the Train and Test sets. The training performance is sometimes slightly lower because there are close to 5 million training programs but we only look at less than 2 million of these programs during training. We sample a subset of only 1000 training programs from the 5 million program set to report the training results in the tables. The test sets also consist of 1000 programs.
6.3 IO2SEQ
In this section, we motivate the use of the R3NN by testing whether a simpler model can also be used to generate programs. The io2seq model is an LSTM whose initial hidden and cell states are a function of the input/output encoding vector. The io2seq model then generates a linearized tree of a program symbol-by-symbol. An example of what a linearized program tree looks like is (S(e(f (ConstStr“@”)ConstStr)f )e)S , which represents the program tree that returns the constant string “@”. Predicting a linearized tree using an LSTM was also done in the context of parsing (Vinyals et al., 2015). For the io2seq model, we used the LSTM-sum cross-correlation I/O conditioning model.
The results in Table 2 show that the performance of the io2seq model at 100 samples per latent test function is far worse than the R3NN, at around 42% versus 91%, respectively. The reasons for that could be that the io2seq model needs to perform far more decisions than the R3NN, since the io2seq model has to predict the parentheses symbols that determine at which level of the tree a particular symbol is at. For example, the io2seq model requires on the order of 100 decisions for length 13 programs, while the R3NN requires no more than 13.
6.4 EFFECT OF SAMPLING MULTIPLE PROGRAMS
For the best R3NN model that we trained, we also evaluated the effect that a different number of samples per latent function had on performance. The results are shown in Table 3. The increase of the model’s performance as the sample size increases hints that the model has a notion of what type of program satisfies a given I/O pair, but it might not be that certain about the details such as which regex to use, etc. By 300 samples, the model is nearing perfect accuracy on the test sets.
6.5 EFFECT OF NUMBER OF INPUT-OUTPUT EXAMPLES
We evaluate the effect of varying the number of input-output examples used to train the Input-output encoders. The 1-best accuracy for train and test data for models trained for 74 epochs is shown in Figure 4. As expected, the accuracy increases with increase in number of input-output examples, since more examples add more information to the encoder and constrain the space of consistent programs in the DSL.
6.6 FLASHFILL BENCHMARKS
We also evaluate our learnt models on 238 real-world FlashFill benchmarks obtained from the Microsoft Excel team and online help-forums. These benchmarks involve string manipulation tasks described using input-output examples. We evaluate two models – one with a cross correlation encoder trained on 5 input-output examples and another trained on 10 input-output examples. Both the models were trained on randomly sampled programs from the DSL upto size 13 with randomly generated input-output examples.
The distribution of the size of smallest DSL programs needed to solve the benchmark tasks is shown in Figure 5(a), which varies from 4 to 63. The figure also shows the number of benchmarks for which our model was able to learn the program using 5 input-output examples using samples of top-2000 learnt programs. In total, the model is able to learn programs for 91 tasks (38.2%). Since the model was trained for programs upto size 13, it is not surprising that it is not able to solve tasks that need larger program size. There are 110 FlashFill benchmarks that require programs upto size 13, out of which the model is able to solve 82.7% of them.
The effect of sampling multiple learnt programs instead of only top program is shown in Figure 5(b). With only 10 samples, the model can already learn about 13% of the benchmarks. We observe a steady increase in performance upto about 2000 samples, after which we do not observe any significant improvement. Since there are more than 2 million programs in the DSL of length 11 itself, the enumerative techniques with uniform search do not scale well (Alur et al., 2015).
We also evaluate a model that is learnt with 10 input-output examples per benchmark. This model can only learn programs for about 29% of the FlashFill benchmarks. Since the FlashFill benchmarks contained only 5 input-output examples for each task, to run the model that took 10 examples as input, we duplicated the I/O examples. Our models are trained on the synthetic training dataset
that is generated uniformly from the DSL. Because of the discrepancy between the training data distribution (uniform) and auxiliary task data distribution, the model with 10 input/output examples might not perform the best on the FlashFill benchmark distribution, even though it performs better on the synthetic data distribution (on which it is trained) as shown in Figure 4.
Our model is able to solve majority of FlashFill benchmarks that require learning programs with upto 3 Concat operations. We now describe a few of these benchmarks, also shown in Figure 6. An Excel user wanted to clean a set of medical billing records by adding a missing “]” to medical codes as shown in Figure 6(a). Our system learns the following program given these 5 input-output examples: Concat(SubStr(v,ConstPos(0),(d,-1,End)), ConstStr(“]”)). The program concatenates the substring between the start of the input string and the position of the last digit regular expression with the constant string “]”. Another task that required user to transform some numbers into a hex format is shown in Figure 6(b). Our system learns the following program: Concat(ConstStr(“0x”),SubStr(v,ConstPos(0),ConstPos(2))). For some benchmarks with long input strings, it is still able to learn regular expressions to extract the desired substring, e.g. it learns a program to extract “NancyF” from the string “123456789,freehafer ,drew ,nancy,19700101,11/1/2007,NancyF@north.com,1230102,123 1st Avenue,Seattle,wa,09999”.
Our system is currently not able to learn programs for benchmarks that require 4 or more Concat operations. Two such benchmarks are shown in Figure 7. The task of combining names in Figure 7(a) requires 6 Concat arguments, whereas the phone number transformation task in Figure 7(b) requires 5 Concat arguments. This is mainly because of the scalability issues in training with programs of larger size. There are also a few interesting benchmarks where the R3NN models gets very close to learning the desired program. For example, for the task “Bill Gates” → “Mr. Bill Gates”, it learns a program that generates “Mr.Bill Gates” (without the whitespace), and for the task “617-444-5454” → “(617) 444-5454”, it learns a program that generates the string “(617 444-5454”.
7 RELATED WORK
We have seen a renewed interest in recent years in the area of Program Induction and Synthesis.
In the machine learning community, a number of promising neural architectures have been proposed to perform program induction. These methods have employed architectures inspired from computation modules (Turing Machines, RAM) (Graves et al., 2014; Kurach et al., 2015; Reed & de Freitas, 2015; Neelakantan et al., 2015) or common data structures such as stacks used in many algorithms (Joulin & Mikolov, 2015). These approaches represent the atomic operations of the network in a differentiable form, which allows for efficient end-to-end training of a neural controller. However, unlike our approach that learns comprehensible complete programs, many of these approaches learn only the program behavior (i.e., they produce desired outputs on new input data). Some recently proposed methods (Kurach et al., 2015; Gaunt et al., 2016; Riedel et al., 2016; Bunel et al., 2016) do learn interpretable programs but these techniques require learning a separate neural network model for each individual task, which is undesirable in many synthesis settings where we would like to learn programs in real-time for a large number of tasks. Liang et al. (2010) restrict the problem space with a probabilistic context-free grammar and introduce a new representation of programs based on combinatory logic, which allows for sharing sub-programs across multiple tasks. They then take a hierarchical Bayesian approach to learn frequently occurring substructures of programs. Our approach, instead, uses neural architectures to condition the search space of programs, and does not require additional step of representing program space using combinatory logic for allowing sharing.
The DSL-based program synthesis approach has also seen a renewed interest recently (Alur et al., 2015). It has been used for many applications including synthesizing low-level bitvector implementations (Solar-Lezama et al., 2005), Excel macros for data manipulation (Gulwani, 2011; Gulwani et al., 2012), superoptimization by finding smaller equivalent loop bodies (Schkufza et al., 2013), protocol synthesis from scenarios (Udupa et al., 2013), synthesis of loop-free programs (Gulwani et al., 2011), and automated feedback generation for programming assignments (Singh et al., 2013). The synthesis techniques proposed in the literature generally employ various search techniques including enumeration with pruning, symbolic constraint solving, and stochastic search, while supporting different forms of specifications including input-output examples, partial programs, program invariants, and reference implementation.
In this paper, we consider input-output example based specification over the hypothesis space defined by a DSL of string transformations, similar to that of FlashFill (without conditionals) (Gulwani, 2011). The key difference between our approach over previous techniques is that our system is trained completely in an end-to-end fashion, while previous techniques require significant manual effort to design heuristics for efficient search. There is some work on guiding the program search using learnt clues that suggest likely DSL expansions, but the clues are learnt over hand-coded textual features of examples (Menon et al., 2013). Moreover, their DSL consists of composition of about 100 high-level text transformation functions such as count and dedup, whereas our DSL consists of tree structured programs over richer regular expression based substring constructs.
There is also a recent line of work on learning probabilistic models of code from a large number of code repositories (big code) (Raychev et al., 2015; Bielik et al., 2016; Hindle et al., 2016), which are then used for applications such as auto-completion of partial programs, inference of variable and method names, program repair, etc. These language models typically capture only the syntactic
properties of code, unlike our approach that also tries to capture the semantics to learn the desired program. The work by Maddison & Tarlow (2014) addresses the problem of learning structured generative models of source code but both their model and application domain are different from ours. Piech et al. (2015) use an NPM-RNN model to embed program ASTs, where a subtree of the AST rooted at a node n is represented by a matrix obtained by combining representations of the children of node n and the embedding matrix of the node n itself (which corresponds to its functional behavior). The forward pass in our R3NN architecture from leaf nodes to the root node is, at a high-level, similar, but we use a distributed representation for each grammar symbol that leads to a different root representation. Moreover, R3NN also performs a reverse-recursive pass to ensure all nodes in the tree encode global information about other nodes in the tree. Finally, the R3NN network is then used to incrementally build a tree to synthesize a program.
The R3NN model employed in our work is related to several tree and graph structured neural networks present in the NLP literature (Le & Zuidema, 2014; Paulus et al., 2014; Irsoy & Cardie, 2013). The Inside-Outside Recursive Neural Network (Le & Zuidema, 2014) in particular is most similar to the R3NN, where they generate a parse tree incrementally by using global leaf-level representations to determine which expansions in the parse tree to take next.
8 CONCLUSION
We have proposed a novel technique called Neuro-Symbolic Program Synthesis that is able to construct a program incrementally based on given input-output examples. To do so, a new neural architecture called Recursive-Reverse-Recursive Neural Network is used to encode and expand a partial program tree into a full program tree. Its effectiveness at example-based program synthesis is demonstrated, even when the program has not been seen during training.
These promising results open up a number of interesting directions for future research. For example, we took a supervised-learning approach here, assuming availability of target programs during training. In some scenarios, we may only have access to an oracle that returns the desired output given an input. In this case, reinforcement learning is a promising framework for program synthesis.
A DOMAIN-SPECIFIC LANGUAGE FOR STRING TRANSFORMATIONS
The semantics of the DSL programs is shown in Figure 8. The semantics of a Concat expression is to concatenate the results of recursively evaluating the constituent substring expressions fi. The semantics of ConstStr(s) is to simply return the constant string s. The semantics of a substring expression is to first evaluate the two position logics pl and pr to p1 and p2 respectively, and then return the substring corresponding to v[p1..p2]. We denote s[i..j] to denote the substring of string s starting at index i (inclusive) and ending at index j (exclusive), and len(s) denotes its length. The semantics of ConstPos(k) expression is to return k if k > 0 or return len + k (if k < 0). The semantics of position logic (r, k, Start) is to return the Start of kth match of r in v from the beginning (if k > 0) or from the end (if k < 0).

The paper presents a method to synthesize string manipulation programs based on a set of input output pairs. The paper focuses on a restricted class of programs based on a simple context free grammar sufficient to solve string manipulation tasks from the FlashFill benchmark. A probabilistic generative model called Recursive-Reverse-Recursive Neural Network (R3NN) is presented that assigns a probability to each program's parse tree after a bottom-up and a top-down pass. Results are presented on a synthetic dataset and a Microsoft Excel benchmark called FlashFill.
The problem of program synthesis is important with a lot of recent interest from the deep learning community. The approach taken in the paper based on parse trees and recursive neural networks seems interesting and promising. However, the model seems too complicated and unclear at several places (details below). On the negative side, the experiments are particularly weak, and the paper does not seem ready for publication based on its experimental results. I was positive about the paper until I realized that the method obtains an accuracy of 38% on FlashFill benchmark when presented with only 5 input-output examples but the performance degrades to 29% when 10 input-output examples are used. This was surprising to the authors too, and they came up with some hypothesis to explain this phenomenon. To me, this is a big problem indicating either a bug in the code or a severe shortcoming of the model. Any model useful for program synthesis needs to be applicable to many input-output examples because most complicated programs require many examples to disambiguate the details of the program.
Given the shortcoming of the experiments, I am not convinced that the paper is ready for publication. Thus, I recommend weak reject. I encourage the authors to address the comments below and resubmit as the general idea seems promising.
More comments:
I am unclear about the model at several places:
- How is the probability distribution normalized? Given the nature of bottom-up top-down evaluation of the potentials, should one enumerate over different completions of a program and the compare their exponentiated potentials? If so, does this restrict the applicability of the model to long programs as the enumeration of the completions gets prohibitively slow?
- What if you only use 1 input-output pair for each program instead of 5? Do the results get better?
- Section 5.1.2 is not clear to me. Can you elaborate by potentially including some examples? Does your input-output representation pre-supposes a fixed number of input-output examples across tasks (e.g. 5 or 10 for all of the tasks)?
Regarding the experiments,
- Could you present some baseline results on FlashFill benchmark based on previous work?
- Is your method only applicable to short programs? (based on the choice of 13 for the number of instructions)
- Does a program considered correct when it is identical to a test program, or is it considered correct when it succeeds on a set of held-out input-output pairs?
- When using 100 or more program samples, do you report the accuracy of the best program out of 100 (i.e. recall) or do you first filter the programs based on training input-output pairs and then evaluate a program that is selected?
Your paper is well beyond the recommended limit of 8 pages. please consider making it shorter.


NATURAL VIDEO SEQUENCE PREDICTION
We propose a deep neural network for the prediction of future frames in natural video sequences. To effectively handle complex evolution of pixels in videos, we propose to decompose the motion and content, two key components generating dynamics in videos. Our model is built upon the Encoder-Decoder Convolutional Neural Network and Convolutional LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. By independently modeling motion and content, predicting the next frame reduces to converting the extracted content features into the next frame content by the identified motion features, which simplifies the task of prediction. Our model is end-to-end trainable over multiple time steps, and naturally learns to decompose motion and content without separate training. We evaluate the proposed network architecture on human activity videos using KTH, Weizmann action, and UCF-101 datasets. We show state-of-the-art performance in comparison to recent approaches. To the best of our knowledge, this is the first end-to-end trainable network architecture with motion and content separation to model the spatio-temporal dynamics for pixel-level future prediction in natural videos.
1 INTRODUCTION
Understanding videos has been one of the most important tasks in the field of computer vision. Compared to still images, the temporal component of videos provides much richer descriptions of the visual world, such as interaction between objects, human activities, and so on. Amongst the various tasks applicable on videos, the task of anticipating the future has recently received increased attention in the research community. Most prior works in this direction focus on predicting high-level semantics in a video such as action (Vondrick et al., 2015; Ryoo, 2011; Lan et al., 2014), event (Yuen and Torralba, 2010; Hoai and Torre, 2013) and motion (Pintea et al., 2014; Walker et al., 2014; Pickup et al., 2014; Walker et al., 2016). Forecasting semantics provides information about what will happen in a video, and is essential to automate decision making. However, the predicted semantics are often specific to a particular task and provide only a partial description of the future. Also, training such models often requires heavily labeled training data which leads to tremendous annotation costs especially with videos.
In this work, we aim to address the problem of prediction of future frames in natural video sequences. Pixel-level predictions provide dense and direct description of the visual world, and existing video recognition models can be adopted on top of the predicted frames to infer various semantics of the future. Spatio-temporal correlations in videos provide a self-supervision for frame prediction, which enables purely unsupervised training of a model by observing raw video frames. Unfortunately, estimating frames is an extremely challenging task; not only because of the inherent uncertainty of the future, but also various factors of variation in videos leading to complicated dynamics in raw pixel values. There have been a number of recent attempts on frame prediction (Srivastava et al., 2015; Mathieu et al., 2015; Oh et al., 2015; Goroshin et al., 2015; Lotter et al., 2015; Ranzato et al., 2014),
∗This work was done while SH and XL were visiting the University of Michigan.
which use a single encoder that needs to reason about all the different variations occurring in videos in order to make predictions of the future, or require extra information like foreground-background segmentation masks and static background (Vondrick et al., 2016).
We propose a Motion-Content Network (MCnet) for robust future frame prediction. Our intuition is to split the inputs for video prediction into two easily identifiable groups, motion and content, and independently capture each information stream with separate encoder pathways. In this architecture, the motion pathway encodes the local dynamics of spatial regions, while the content pathway encodes the spatial layout of the salient parts of an image. The prediction of the future frame is then achieved by transforming the content of the last observed frame given the identified dynamics up to the last observation. Somewhat surprisingly, we show that such a network is end-to-end trainable without individual path way supervision. Specifically, we show that an asymmetric architecture for the two pathways enables such decompositions without explicit supervision. The contributions of this paper are summarized below:
• We propose MCnet for the task of frame prediction, which separates the information streams (motion and content) into different encoder pathways.
• The proposed network is end-to-end trainable and naturally learns to decompose motion and content without separate training, and reduces the task of frame prediction to transforming the last observed frame into the next by the observed motion.
• We evaluate the proposed model on challenging real-world video datasets, and show that it outperforms previous approaches on frame prediction.
The rest of the paper is organized as follows. We briefly review related work in Section 2, and introduce an overview of the proposed algorithm in Section 3. The detailed configuration of the proposed network is described in Section 4. Section 5 describes training and inference procedure. Section 6 illustrates implementation details and experimental results on challenging benchmarks.
2 RELATED WORK
The problem of visual future prediction has received growing interests in the computer vision community. It has led to various tasks depending on the objective of future prediction, such as human activity (Vondrick et al., 2015; Ryoo, 2011; Lan et al., 2014), event (Yuen and Torralba, 2010; Hoai and Torre, 2013) and geometric path (Walker et al., 2014). Although previous work achieved reasonable success in specific tasks, they are often limited to estimating predefined semantics, and require fully-labeled training data. To alleviate this issue, approaches predicting representation of the future beyond semantic labels have been proposed. Walker et al. (2014) proposed a data-driven approach to predict the motion of a moving object, and coarse hallucination of the predicted motion. Vondrick et al. (2015) proposed a deep regression network to predict feature representations of the future frames. These approaches are supervised and provide coarse predictions of how the future will look like. Our work also focuses on unsupervised learning for prediction of the future, but to a more direct visual prediction task: frame prediction.
Compared to predicting semantics, pixel-level prediction has been less investigated due to the difficulties in modeling evolution of raw pixels over time. Fortunately, recent advances in deep learning provide a powerful tool for sequence modeling, and enable the creation of novel architectures for modeling complex sequential data. Ranzato et al. (2014) applied a recurrent neural network developed for language modeling to frame prediction by posing the task as classification of each image region to one of quantized patch dictionaries. Srivastava et al. (2015) applied a sequence-tosequence model to video prediction, and showed that Long Short-Term Memory (LSTM) is able to capture pixel dynamics. Oh et al. (2015) proposed an action-conditional encoder-decoder network to predict future frames in Atari games. In addition to the different choices of architecture, some other works addressed the importance of selecting right objective function: Lotter et al. (2015) used adversarial loss with combined CNN and LSTM architectures, and Mathieu et al. (2015) employed similar adversarial loss with additional regularization using a multi-scale encoder-decoder network. Finn et al. (2016) constructed a network that predicts transformations on the input pixels for next frame prediction. Patraucean et al. (2015) proposed a network that by explicitly predicting optical flow features is able to predict the next frame in a video. Vondrick et al. (2016) proposed a generative adversarial network for video which, by generating a background-foreground mask, is able to generate
realistic-looking video sequences. However, none of the previously mentioned approaches exploit spatial and temporal information separately in an unsupervised fashion. In terms of the way data is observed, the closest work to ours is Xue et al. (2016). The differences are (1) Our model is deterministic and theirs is probabilistic, (2) our motion encoder is based on convolutional LSTM (Shi et al., 2015) which is a more natural module to model long-term dynamics, (3) our content encoder observes a single scale input and theirs observes many scales, and (4) we directly generate image pixels values, which is a more complicated task. We aim to exploit the existing spatio-temporal correlations in videos by decomposing the motion and content in our network architecture.
To the best of our knowledge, the idea of separating motion and content has not been investigated in the task of unsupervised deterministic frame prediction. The proposed architecture shares similarities to the two-stream CNN (Simonyan and Zisserman, 2014), which is designed for action recognition to jointly exploit the information from frames and their temporal dynamics. However, in contrast to their network we aim to learn features for temporal dynamics directly from the raw pixels, and we use the identified features from the motion in combination with spatial features to make pixel-level predictions of the future.
3 ALGORITHM OVERVIEW
In this section, we formally define the task of frame prediction and the role of each component in the proposed architecture. Let xt ∈ Rw×h×c denote the t-th frame in an input video x, where w, h, and c denote width, height, and number of channels, respectively. The objective of frame prediction is to generate the future frame x̂t+1 given the input frames x1:t.
At the t-th time step, our network observes a history of previous consecutive frames up to frame t, and generates the prediction of the next frame x̂t+1 as follows:
• Motion Encoder recurrently takes an image difference input between frame xt and xt−1 starting from t = 2, and produces the hidden representation dt encoding the temporal dynamics of the scene components (Section 4.1). • Content Encoder takes the last observed frame xt as an input, and outputs the hidden
representation st that encodes the spatial layout of the scene (Section 4.2). • Multi-Scale Motion-Content Residual takes the computed features, from both the motion
and content encoders, at every scale right before pooling and computes residuals rt (He et al., 2015) to aid the information loss caused by pooling in the encoding phase (Section 4.3). • Combination Layers and Decoder takes the outputs from both encoder pathways and
residual connections, dt, st, and rt, and combines them to produce a pixel-level prediction of the next frame x̂t+1 (Section 4.4).
The overall architecture of the proposed algorithm is described in Figure 1. The prediction of multiple frames, x̂t+1:t+T , can be achieved by recursively performing the above procedures over T time steps (Section 5). Each component in the proposed architecture is described in the following section.
4 ARCHITECTURE
This section describes the detailed configuration of the proposed architecture, including the two encoder pathways, multi-scale residual connections, combination layers, and decoder.
4.1 MOTION ENCODER
The motion encoder captures the temporal dynamics of the scene’s components by recurrently observing subsequent difference images computed from xt−1 and xt, and outputs motion features by
[dt, ct] = f dyn (xt − xt−1,dt−1, ct−1) , (1)
where xt−xt−1 denotes element-wise subtraction between frames at time t and t−1, dt ∈ Rw ′×h′×c′ is the feature tensor encoding the motion across the observed difference image inputs, and ct ∈ Rw′×h′×c′ is a memory cell that retains information of the dynamics observed through time. f dyn is implemented in a fully-convolutional way to allow our model to identify local dynamics of frames
64 64
64 64
rather than complicated global motion. For this, we use an encoder CNN with a Convolutional LSTM (Shi et al., 2015) layer on top.
4.2 CONTENT ENCODER
The content encoder extracts important spatial features from a single frame, such as the spatial layout of the scene and salient objects in a video. Specifically, it takes the last observed frame xt as an input, and produces content features by
st = f cont (xt) , (2)
where st ∈ Rw ′×h′×c′ is the feature encoding the spatial content of the last observed frame, and f cont is implemented by a Convolutional Neural Network (CNN) that specializes on extracting features from single frame.
It is important to note that our model employs an asymmetric architecture for the motion and content encoder. The content encoder takes the last observed frame, which keeps the most critical clue to reconstruct spatial layout of near future, but has no information about dynamics. On the other hand, the motion encoder takes a history of previous image differences, which are less informative about the future spatial layout compared to the last observed frame, yet contain important spatio-temporal variations occurring over time. This asymmetric architecture encourages encoders to exploit each of two pieces of critical information to predict the future content and motion individually, and enables the model to learn motion and content decomposition naturally without any supervision.
4.3 MULTI-SCALE MOTION-CONTENT RESIDUAL
To prevent information loss after the pooling operations in our motion and content encoders, we use residual connections (He et al., 2015). The residual connections in our network communicate motion-content features at every scale into the decoder layers after unpooling operations. The residual feature at layer l is computed by
rlt = f res ([slt,dlt])l , (3)
where rlt is the residual output at layer l, [ slt,d l t ] is the concatenation of the motion and content features along the depth dimension at layer l of their respective encoders, f res (.)l the residual function at layer l implemented as consecutive convolution layers and rectification with a final linear layer.
4.4 COMBINATION LAYERS AND DECODER
The outputs from the two encoder pathways, dt and st, encode a high-level representation of motion and content, respectively. Given these representations, the objective of the decoder is to generate a
pixel-level prediction of the next frame x̂t+1 ∈ Rw×h×c. To this end, it first combines the motion and content back into a unified representation by
ft = g comb ([dt, st]) , (4)
where [dt, st] ∈ Rw ′×h′×2c′ denotes the concatenation of the higher-level motion and content features in the depth dimension, and ft ∈ Rw ′×h′×c′ denotes the combined high-level representation of motion and content. gcomb is implemented by a CNN with bottleneck layers (Hinton and Salakhutdinov, 2006); it first projects both dt and st into a lower-dimensional embedding space, and then puts it back to the original size to construct the combined feature ft. Intuitively, ft can be viewed as the content feature of the next time step, st+1, which is generated by transforming st using the observed dynamics encoded in dt. Then our decoder places ft back into the original pixel space by
x̂t+1 = g dec (ft, rt) , (5)
where rt is a list containing the residual connections from every layer of the motion and content encoders before pooling sent to every layer of the decoder after unpooling. We employ the deconvolution network (Zeiler et al., 2011) for our decoder network gdec, which is composed of multiple successive operations of deconvolution, rectification and unpooling with the addition of the motioncontent residual connections after each unpooling operation. The output layer is passed through a tanh (.) activation function. Unpooling with fixed switches are used to upsample the intermediate activation maps.
5 INFERENCE AND TRAINING
Section 4 describes the procedures for single frame prediction, while this section presents the extension of our algorithm for the prediction of multiple time steps.
5.1 MULTI-STEP PREDICTION
Given an input video, our network observes the first n frames as image difference between frame xt and xt−1, starting from t = 2 up to t = n, to encode initial temporal dynamics through the motion encoder. The last frame xn is given to the content encoder to be transformed into the first prediction x̂t+1 by the identified motion features.
For each time step t ∈ [n+ 1, n+ T ], where T is the desired number of prediction steps, our network takes the difference image between the first prediction x̂t+1 and the previous image xt, and the first prediction x̂t+1 itself to predict the next frame x̂t+2, and so forth.
5.2 TRAINING OBJECTIVE
To train our network, we use an objective function composed of different sub-losses similar to Mathieu et al. (2015). Given the training data D = {x(i)1,...,T }Ni=1, our model is trained to minimize the prediction loss by L = αLimg + βLGAN, (6) where α and β are hyper-parameters that control the effect of each sub-loss during optimization. Limg is the loss in image space from Mathieu et al. (2015) defined by
Limg = Lp (xt+k, x̂t+k) + Lgdl (xt+k, x̂t+k) , (7)
where Lp (y, z) = T∑ k=1 ||y − z||pp, (8)
Lgdl (y, z) = h,w∑ i,j | (|yi,j − yi−1,j | − |zi,j − zi−1,j |) |λ (9)
+ | (|yi,j−1 − yi,j | − |zi,j−1 − zi,j |) |λ . Here, xt+k and x̂t+k are the target and predicted frames, respectively, and p and λ are hyperparameters forLp andLgdl, respectively. Intuitively,Lp guides our network to match the average pixel
values directly, while Lgdl guides our network to match the gradients of such pixel values. Overall, Limg guides our network to learn parameters towards generating the correct average sequence given the input. Training to generate average sequences, however, results in somewhat blurry generations which is the reason we use an additional sub-loss. LGAN is the generator loss in adversarial training to allow our model to predict realistic looking frames and it is defined by
LGAN = − logD ([x1:t, G (x1:t)]) , (10) where x1:t is the concatenation of the input images, xt+1:t+T is the concatenation of the ground-truth future images, G (x1:t) = x̂t+1:t+T is the concatenation of all predicted images along the depth dimension, andD (.) is the discriminator in adversarial training. The discriminative loss in adversarial training is defined by
Ldisc = − logD ([x1:t,xt+1:t+T ])− log (1−D ([x1:t, G (x1:t)])) . (11) LGAN, in addition to Limg, allows our network to not only generate the target sequence, but also simultaneously enforce realism in the images through visual sharpness that fools the human eye. Note that our model uses its predictions as input for the next time-step during the training, which enables the gradients to flow through time and makes the network robust for error propagation during prediction. For more a detailed description about adversarial training, please refer to Appendix D.
6 EXPERIMENTS
In this section, we present experiments using our network for video generation. We first evaluate our network, MCnet, on the KTH (Schuldt et al., 2004) and Weizmann action (Gorelick et al., 2007) datasets, and compare against a baseline convolutional LSTM (ConvLSTM) (Shi et al., 2015). We then proceed to evaluate on the more challenging UCF-101 (Soomro et al., 2012) dataset, in which we compare against the same ConvLSTM baseline and also the current state-of-the-art method by Mathieu et al. (2015). For all our experiments, we use α = 1, λ = 1, and p = 2 in the loss functions.
In addition to the results in this section, we also provide more qualitative comparisons in the supplementary material and in the videos on the project website: https://sites.google. com/a/umich.edu/rubenevillegas/iclr2017.
Architectures. The content encoder of MCnet is built with the same architecture as VGG16 (Simonyan and Zisserman, 2015) up to the third pooling layer. The motion encoder of MCnet is also similar to VGG16 up to the third pooling layer, except that we replace its consecutive 3x3 convolutions with single 5x5, 5x5, and 7x7 convolutions in each layer. The combination layers are composed of 3 consecutive 3x3 convolutions (256, 128, and 256 channels in each layer). The multi-scale residuals are composed of 2 consecutive 3x3 convolutions. The decoder is the mirrored architecture of the content encoder where we perform unpooling followed by deconvolution. For the baseline ConvLSTM, we use the same architecture as the motion encoder, residual connections, and decoder, except we increase the number of channels in the encoder in order to have an overall comparable number of parameters with MCnet.
6.1 KTH AND WEIZMANN ACTION DATASETS
Experimental settings. The KTH human action dataset (Schuldt et al., 2004) contains 6 categories of periodic motions on a simple background: running, jogging, walking, boxing, hand-clapping and hand-waiving. We use person 1-16 for training and 17-25 for testing, and also resize frames to 128x128 pixels. We train our network and baseline by observing 10 frames and predicting 10 frames into the future on the KTH dataset. We set β = 0.02 for training. We also select the walking, running, one-hand waving, and two-hands waving sequences from the Weizmann action dataset (Gorelick et al., 2007) for testing the networks’ generalizability.
For all the experiments, we test the networks on predicting 20 time steps into the future. As for evaluation, we use the same SSIM and PSNR metrics as in Mathieu et al. (2015). The evaluation on KTH was performed on sub-clips within each video in the testset. We sample sub-clips every 3 frames for running and jogging, and sample sub-clips every 20 frames (skipping the frames we have already predicted) for walking, boxing, hand-clapping, and hand-waving. Sub-clips for running, jogging, and walking were manually trimmed to ensure humans are always present in the frames. The evaluation on Weizmann was performed on all sub-clips in the selected sequences.
Results. Figure 2 summarizes the quantitative comparisons among our MCnet, ConvLSTM baseline and their residual variations. In the KTH test set, our network outperforms the ConvLSTM baseline by a small margin. However, when we test the residual versions of MCnet and ConvLSTM on the dataset (Gorelick et al., 2007) with similar motions, we can see that our network can generalize well to the unseen contents by showing clear improvements, especially in long-term prediction. One reason for this result is that the test and training partitions of the KTH dataset have simple and similar image contents so that ConvLSTM can memorize the average background and human appearance to make reasonable predictions. However, when tested on unseen data, ConvLSTM has to internally take care of both scene dynamics and image contents in a mingled representation, which gives it a hard time for generalization. In contrast, the reason our network outperforms the ConvLSTM baseline on unseen data is that our network focuses on identifying general motion features and applying them to a learned content representation.
Figure 3 presents qualitative results of multi-step prediction by our network and ConvLSTM. As expected, prediction results by our full architecture preserves human shapes more accurately than the baseline. It is worth noticing that our network produces very sharp prediction over long-term time steps; it shows that MCnet is able to capture periodic motion cycles, which reduces the uncertainty of future prediction significantly. More qualitative comparisons are shown in the supplementary material and the project website.
6.2 UCF-101 DATASET
Experimental settings. This section presents results on the challenging real-world videos in the UCF-101 (Soomro et al., 2012) dataset. Having collected from YouTube, the dataset contains 101 realistic human actions taken in a wild and exhibits various challenges, such as background clutter, occlusion, and complicated motion. We employed the same network architecture as in the KTH dataset, but resized frames to 240x320 pixels, and trained the network to observe 4 frames and predict a single frame. We set β = 0.001 for training. We also trained our convolutional LSTM baseline in the same way. Following the same protocol as Mathieu et al. (2015) for data pre-processing and
evaluation metrics on full images, all networks were trained on Sports-1M (Karpathy et al., 2014) dataset and tested on UCF-101 unless otherwise stated.1
Results. Figure 4 shows the quantitative comparisons between our network trained for single-stepprediction and Mathieu et al. (2015). We can clearly see the advantage of our network over the baseline. The separation of motion and contents in two encoder pathways allows our network to identify key motion and content features, which are then fed into the decoder to yield predictions of higher quality compared to the baseline.2 In other words, our network only moves what shows motion in the past, and leaves the rest untouched. We also trained a residual version of MCnet on UCF-101, indicated by “MCnet + RES UCF101", to compare how well our model generalizes when trained and tested on the same or different dataset(s). To our surprise, when tested with UCF-101, the MCnet trained on Sports-1M (MCnet + RES) roughly matches the performance of the MCnet trained on UCF-101 (MCnet + RES UCF101), which suggests that our model learns effective representations which can generalize to new datasets. Figure 5 presents qualitative comparisons between frames generated by our network and Mathieu et al. (2015). Since the ConvLSTM and Mathieu et al. (2015) lack explicit motion and content modules, they lose sense of the dynamics in the video and therefore the contents become distorted quickly. More qualitative comparisons are shown in the supplementary material and the project website.
1We use the code and model released by Mathieu et al. (2015) at https://github.com/coupriec/ VideoPredictionICLR2016
2We were not able to get the model fine-tuned on UCF-101 from the authors so it is not included in Figure 4
7 CONCLUSION
We proposed a motion-content network for pixel-level prediction of future frames in natural video sequences. The proposed model employs two separate encoding pathways, and learns to decompose motion and content without explicit constraints or separate training. Experimental results suggest that separate modeling of motion and content improves the quality of the pixel-level future prediction, and our model overall achieves state-of-the-art performance in predicting future frames in challenging real-world video datasets.
8 ACKNOWLEDGEMENTS
This work was supported in part by ONR N00014-13-1-0762, NSF CAREER IIS-1453651, gifts from the Bosch Research and Technology Center, and Sloan Research Fellowship. We also thank NVIDIA for donating K40c and TITAN X GPUs. We thank Ye Liu, Junhyuk Oh, Xinchen Yan, Lajanugen Logeswaran, Yuting Zhang, Sungryull Sohn, Kibok Lee, Rui Zhang, and other collaborators for helpful discussions. R. Villegas was partly supported by the Rackham Merit Fellowship.
9 APPENDIX
A QUALITATIVE AND QUANTITATIVE COMPARISON WITH CONSIDERABLE CAMERA MOTION AND ANALYSIS
In this section, we show frame prediction examples in which considerable camera motion occurs. We analyze the effects of camera motion on our best network and the corresponding baselines. First, we analyze qualitative examples on UCF101 (more complicated camera motion) and then on KTH (zoom-in and zoom-out camera effect).
UCF101 Results. As seen in Figure 9 and Figure 10, our model handles foreground and camera motion for a few steps. We hypothesize that for the first few steps, motion signals from images are clear. However, as images are predicted, motion signals start to deteriorate due to prediction errors. When a considerable amount of camera motion is present in image sequences, the motion signals are very dense. As predictions evolve into the future, our motion encoder has to handle large motion deterioration due to prediction errors, which cause motion signals to get easily confused and lost quickly.
KTH Results. We were unable to find videos with background motion in the KTH dataset, but we found videos where the camera is zooming in or out for the actions of boxing, handclapping, and handwaving. In Figure 11, we display qualitative for such videos. Our model is able to predict the zoom change in the cameras, while continuing the action motion. In comparison to the performance observed in UCF101, the background does not change much. Thus, the motion signals are well localized in the foreground motion (human), and do not get confused with the background and lost as quickly.
B EXTENDED QUANTITATIVE EVALUATION
In this section, we show additional quantitative comparison with a baseline based on copying the last observed frame through time for KTH and UCF101 datasets. Copying the last observed frame through time ensures perfect background prediction in videos where most of the motion comes from foreground (i.e. person performing an action). However, if such foreground composes a small part of the video, it will result in high prediction quality score regardless of the simple copying action.
In Figure 12 below, we can see the quantitative comparison in the datasets. Copying the last observed frame through time does a reasonable job in both datasets, however, the impact is larger in UCF101. Videos in the KTH dataset comprise simple background with minimal camera motion, which allows our network to easily predict both foreground and background motion, resulting in better image quality scores. However, videos in UCF101 contain more complicated and diverse background which in combination with camera motion present a much greater challenge to video prediction networks. From the qualitative results in Section A and Figures 5, 8, 9, and 10, we can see that our network performs better in videos that contain isolated areas of motion compared to videos with dense motion. A simple copy/paste operation of the last observed frame, ensures very high prediction scores in videos where very small motion occur. The considerable score boost by videos with small motion causes the simple copy/paste baseline to outperform MCnet in the overall performance on UCF101.
C UCF101 MOTION DISAMBIGUATION EXPERIMENTS
Due to the observed bias from videos with small motion, we perform experiments by measuring the image quality scores on areas of motion. These experiments are similar to the ones performed in Mathieu et al. (2015). We compute DeepFlow optical flow (Weinzaepfel et al., 2013) between the previous and the current groundtruth image of interest, compute the magnitude, and normalize it to [0, 1]. The computed optical flow magnitude is used to mask the pixels where motion was observed. We set the pixels where the optical flow magnitude is less than 0.2, and leave all other pixels untouched in both the groundtruth and predicted images. Additionally, we separate the test videos by the average `2-norm of time difference between target frames. We separate the test videos into deciles based of the computed average `2-norms, and compute image quality on each decile. Intuitively, the 1st decile contains videos with the least overall of motion (i.e. frames that show the smallest change over time), and the 10th decile contains videos with the most overall motion (i.e. frames that show the largest change over time).
As shown in Figure 13, when we only evaluate on pixels where rough motion is observed, MCnet reflects higher PSNR and SSIM, and clearly outperforms all the baselines in terms of SSIM. The SSIM results show that our network is able to predict a structure (i.e. textures, edges, etc) similar to the grountruth images within the areas of motion. The PSNR results, however, show that our method outperforms the simple copy/paste baseline for the first few steps, but then our method performs slightly worse. The discrepancies observed between PSNR and SSIM scores could be due to the fact that some of the predicted images may not reflect the exact pixel values of the groundtruth regardless of the structures being similar. SSIM scores are known to take into consideration features in the image that go beyond directly matching pixel values, reflecting more accurately how humans perceived image quality.
Figures 15 and 14 show the evaluation by separating the test videos into deciles based on the average `2-norm of time difference between target frames. From this evaluation, it is proven that the copy last frame baseline scores higher in videos where motion is the smallest. The first few deciles (videos with small motion) show that our network is not just copying the last observed frame through time, otherwise it would perform similarly to the copy last frame baseline. The last deciles (videos with large motion) show our network outperforming all the baselines, including the copy last frame baseline, effectively confirming that our network does predict motion similar to the motion observed in the video.
D ADVERSARIAL TRAINING
Mathieu et al. (2015) proposed an adversarial training for frame prediction. Inspired by Goodfellow et al. (2014), they proposed a training procedure that involves a generative model G and a discriminative model D. The two models compete in a two-player minimax game. The discriminator D is optimized to correctly classify its inputs as either coming from the training data (real frame sequence) or from the generator G (synthetic frame sequence). The generator G is optimized to generate frames that fool the discriminator into believing that they come from the training data. At training time, D takes the concatenation of the input frames that go into G and the images produced by G. The adversarial training objective is defined as follows:
min G max D
logD ([x1:t,xt+1:t+T ]) + log (1−D ([x1:t, G (x1:t)])) ,
where [., .] denotes concatenation in the depth dimension, x1:t denotes the input frames toG, xt+1:t+T are the target frames, andG (x1:t) = x̂t+1:t+T are the frames predicted byG. In practice, we split the minimax objective into two separate, but equivalent, objectives: LGAN and Ldisc. During optimization, we minimize the adversarial objective alternating between LGAN and Ldisc. LGAN is defined by
LGAN = − logD ([x1:t, G (x1:t)]) ,
where we optimize the parameters of G to minimize LGAN while the parameters of D stay untouched. As a result, G is optimized to generate images that make D believe that they come from the training data. Thus, the generated images look sharper, and more realistic. Ldisc is defined by
Ldisc = − logD ([x1:t,xt+1:t+T ])− log (1−D ([x1:t, G (x1:t)])) ,
where we optimize the parameters of D to minimize Ldisc, while the parameters of G stay untouched. D tells us whether its input came from the training data or the generator G. Alternating between the two objectives, causes G to generate very realistic images, and D not being able to distinguish between generated frames and frames from the training data.

1) Summary
This paper investigates the usefulness of decoupling appearance and motion information for the problem of future frame prediction in natural videos. The method introduces a novel two-stream encoder-decoder architecture, MCNet, consisting of two separate encoders -- a convnet on single frames and a convnet+LSTM on sequences of temporal differences -- followed by combination layers (stacking + convolutions) and a deconvolutional network decoder leveraging also residual connections from the two encoders. The architecture is trained end-to-end using the objective and adversarial training strategy of Mathieu et al.
2) Contributions
+ The architecture seems novel and is well motivated. It is also somewhat related to the two-stream networks of Simonyan & Zisserman, which are very effective for real-world action recognition.
+ The qualitative results are numerous, insightful, and very convincing (including quantitatively) on KTH & Weizmann, showing the benefits of decoupling content and motion for simple scenes with periodic motions, as well as the need for residual connections.
3) Suggestions for improvement
Static dataset bias:
In response to the pre-review concerns about the observed static nature of the qualitative results, the authors added a simple baseline consisting in copying the pixels of the last observed frame. On the one hand, the updated experiments on KTH confirm the good results of the method in these conditions. On the other hand, the fact that this baseline is better than all other methods (not just the authors's) on UCF101 casts some doubts on whether reporting average statistics on UCF101 is insightful enough. Although the authors provide some qualitative analysis pertaining to the quantity of motion, further quantitative analysis seems necessary to validate the performance of this and other methods on future frame prediction. At least, the results on UCF101 should be disambiguated with respect to the type of scene, for instance by measuring the overall quantity of motion (e.g., l2 norm of time differences) and reporting PSNR and SSIM per quartile / decile. Ideally, other realistic datasets than UCF101 should be considered in complement. For instance, the Hollywood 2 dataset of Marszalek et al would be a good candidate, as it focuses on movies and often contains complex actor, camera, and background motions that would make the "pixel-copying" baseline very poor. Experiments on video datasets beyond actions, like the KITTI tracking benchmark, would also greatly improve the paper.
Additional recognition experiments:
As mentioned in pre-review questions, further UCF-101 experiments on action recognition tasks by fine-tuning would also greatly improve the paper. Classifying videos indeed requires learning both appearance and motion features, and the two-stream encoder + combination layers of the MCNet+Res architecture seem particularly adapted, if they indeed allowed for unsupervised pre-trainining of content and motion representations, as postulated by the authors. These experiments would also contribute to dispelling the aforementioned concerns about the static nature of the learned representations.
4) Conclusion
Overall, this paper proposes an interesting architecture for an important problem, but requires additional experiments to substantiate the claims made by the authors. If the authors make the aforementioned additional experiments and the results are convincing, then this paper would be clearly relevant for ICLR.
5) Post-rebuttal final decision
The authors did a significant amount of additional work, following the suggestions made by the reviewers, and providing additional compelling experimental evidence. This makes this one of the most experimentally thorough ones for this problem. I, therefore, increase my rating, and suggest to accept this paper. Good job!