Automatic Program Synthesis of Long Programs with a Learned Garbage Collector
We consider the problem of generating automatic code given sample input-output pairs. We train a neural network to map from the current state and the outputs to the program's next statement. The neural network optimizes multiple tasks concurrently: the next operation out of a set of high level commands, the operands of the next statement, and which variables can be dropped from memory. Using our method we are able to create programs that are more than twice as long as existing state-of-the-art solutions, while improving the success rate for comparable lengths, and cutting the run-time by two orders of magnitude. Our code, including an implementation of various literature baselines, is publicly available at https: //github.com/amitz25/PCCoder
This paper presents an approach to use neural networks to guide the search of programs in a DSL to perform program synthesis from few input-output examples. In particular, it learns two networks f and g to predict the next statement and which variables to drop from the memory, given the current state of the program together with the desired output. During training, an auxiliary task of predicting operators (with lambda functions) is used to improve the training. The environment state is represented as a set of fixed dimensional embeddings, which are then passed through a dense block and pooled to obtain example order-independent representation. This representation is then used to compute three predictions: a multiclass statement prediction f, a multi-label binary classification g, and an auxiliary task of predicting statement operators and the lambda functions. The search is performed using the CAB extension for beam search that iteratively weakens the pruning heuristics to perform new beam search runs. The technique is evaluated on the DeepCoder dataset and is shown to significantly outperform DeepCoder and baselines both in terms of synthesis time and also in terms of the lengths of the programs that can be synthesized.  Overall, this paper presents a nice idea to use program states (dynamic environments) for predicting next functions. The technique is well-presented and a comprehensive evaluation is performed with many different ablation settings to evaluate the usefulness of different design choices. The results in comparison to previous techniques like DeepCoder are quite impressive and I also liked various ablation experiments.  The paper is mostly well written and clear, but the model description parts can be presented more clearly. The evaluation is described in detail with reasonable design choices. The idea of using dynamic state information and variable dropping for learning to search in synthesis is novel. The technique is evaluated on toy-ish integer benchmarks (unlike RobustFill or Karel with real-world synthesis benchmarks), but still can be useful for making progress on hard synthesis tasks.  Given the current progress in neural program synthesis approaches, DeepCoder is quite a weak baseline now. Note that DeepCoder trains only example embeddings and doesn’t train the search component, and relies on an external search. In particular, with systems like RobustFill, where one trains the search as well during training would be a better baseline method to compare here. Having said that the paper makes a key contribution here of using dynamic program states for making predictions, unlike RobustFill that only uses the syntactic partial programs. One way to perform such a comparison can be to train PCCoder with environment embeddings consisting of only (input-output) states without the additional intermediate variables. That would clarify precisely the contribution of having dynamic program states during the prediction process.  The model description for dense blocks is also not described in great detail and I have several questions about the model’s design choices. Since the dense blocks seem to have a rather significant impact on performance (73% vs 83%), it would be good to understand the key properties of such a network compared to previous models used for synthesis.  As the model is performing pooling over the environment state embeddings, is it the case that it can be provided an arbitrary number of input-output examples at test time?  What is the size of the possible statements |S|? With increasing number of choices for operator arguments in a more complex DSL, this set can easily blow up. Why not use different networks for predicting functions and operands? It seems the auxiliary task of predicting functions is anyways being helpful. What happens if one adds other auxiliary task of predicting operands?  It was also not clear what is the complexity of the programs that PCCoder can currently synthesize. In the author response, it would be great if the authors can provide a few samples of length 12 and length 14 programs that PCCoder can synthesize together with the corresponding input-output examples. In figure 4, what are the 8 different programs and the environments?  For embedding program states, the current approach uses a relatively simple concatenation of variable embeddings. It might be interesting to explore other ways to embed program states such as the ones suggested in [1].   1. Dynamic Neural Program Embedding for Program Repair. Ke Wang, Rishabh Singh, Zhendong Su. ICLR 2018