AIN'T NOBODY GOT TIME FOR CODING: STRUCTURE-AWARE PROGRAM SYNTHESIS FROM NATURAL LANGUAGE
Program synthesis from natural language (NL) is practical for humans and, once technically feasible, would significantly facilitate software development and revolutionize end-user programming. We present SAPS, an end-to-end neural network capable of mapping relatively complex, multi-sentence NL specifications to snippets of executable code. The proposed architecture relies exclusively on neural components, and is trained on abstract syntax trees, combined with a pretrained word embedding and a bi-directional multi-layer LSTM for processing of word sequences. The decoder features a doubly-recurrent LSTM, for which we propose novel signal propagation schemes and soft attention mechanism. When applied to a large dataset of problems proposed in a previous study, SAPS performs on par with or better than the method proposed there, producing correct programs in over 92% of cases. In contrast to other methods, it does not require post-processing of the resulting programs, and uses a fixed-dimensional latent representation as the only interface between the NL analyzer and the source code generator.
# Summary
This paper introduces a model called SAPS for the task of mapping natural language descriptions of programs to the AST tree of the corresponding program. The model consists of a variation of a double recurrent neural network (DRNN) which is pre-trained using an autoencoder. The natural language description is turned into a latent vector using pretrained word embeddings and a bidirectional stacked LSTM. The final model consists of training this sentence embedding model jointly with the decoder of the autoencoder.
# Quality
The authors introduce a reasonable model which achieves good performance on a relevant task. The results section contains a fair amount of numbers which give some insight into the performance of the model. However, the results make it hard to compare the proposed model with other models, or with other training procedures.
For example, the Seq2Tree model that is shown in table 2 was not necessarily intended to be used without a search algorithm. It is also not mentioned how many parameters both models have which makes it hard to judge how fair of a comparison it is. (I couldn't find the dimensionality of the encoder in the text, and the decoder dimensionality is only shown in figure 2.)
The model proposed in this work uses decoder pretrained in an autoencoder setting. No results are shown for how the model performs without pretraining. Pretraining using autoencoders is a technique that fell out of favor years ago, so it seems worthwhile to investigate whether or not this pretraining is necessary, and if so, why and how it aids the final performance.
It is unclear to me what type of robustness the authors are trying to show in table 5. The use of robustness here is not clear (robustness is often used to refer to a network's susceptability to adversarial attacks or perturbations of the weights). It also seems that the type of "simple replacements" mentioned are very similar to the way the examples were generated in the first place (section 4 of Polosukhin). If the goal is to measure generalization, why do the authors believe that performance on the test set alone is not a sufficient measure of generalization?
Some smaller comments and questions:
* In section 4 you mention training the sentence-to-tree and the sentence-to-vector mappings. Isn't the sentence-to-vector model a subset of the sentence-to-tree model? Should I interpret this as saying that, given the pretrained decoder and the glove embeddings, you now train the entire model jointly? Or do you first learn the mapping from sentences to the latent space, and only then finetune the entire model?
* The attention mechanism is not actually an attention mechanism: Attention mechanisms are used to reduce a variable number of elements to a single element by learning a weighting function and taking a weighted sum. My understanding is that in this case, the input (the latent representation) is of fixed size. The term "gating function" would be more appropriate.
* You specify that the hidden states of the decoder are initialized to zero, but don't specify what the cell states are initialized to.
# Clarity
The writing in the paper is passable. It lacks a bit in structure (e.g., I would introduce the problem and dataset before introducing the model) and sometimes fails to explain what insights the authors draw from certain results, or why certain results are reported. Take table 3 as an example: As a reader, I was confused at first why I should care about the reconstruction performance of the autoencoder alone, considering its only purpose is pretraining. Then, when looking at the numbers, I am even more confused since it is counterintuitive that it is harder to copy a program than it is to infer it. At the end of the paragraph the authors propose an explanation (the encoder isn't as powerful as the decoder) but leave it unclear as to why these numbers were being reported in the first place.
In general, the paper would do well to restructure the text so that the reader is explained what the goal of the different experiments is, and what insights should be drawn from them.
A variety of smaller concerns and comments:
* Please reduce and harmonize the terminology in the paper: the terms latent-to-AST, NLP2Tree, NLP2Vec, tree2tree/tree-to-tree, sentence-to-tree, sentence-to-vector, NL-to-latent, and spec-to-latent all appear in the paper and several of them are redundant, making it significantly harder to follow along with the text.
* Avoid citing the same work multiple times within a paragraph; cite only the first use and use prose to make clear that future references are to the same work.
* Formula 14 has h_i^{(pred)} on both sides of the quation, and is used in the definition of A as well. I am assuming these two terms are actually the h_i^{(pred)} from equation 8, but this should be made clear in the notation.
* Why does figure 1 have boxes for "NL specification", "NL spec.", and "NL query"? In general, the boxes inconsistently seem to represent both values and operations.
* It is never explicitly stated that Seq2Tree is the model from the Polosukhin et al. paper, which is a bit confusing.
* Parameters is missing an -s in the first paragraph of section 4.
* It is said that regularization is applied to layer normalization, which I assume means that the regularization is applied to the gain parameters of the layer normalization.
* It says "like a in the above example" when table 1 is rendered at the bottom of the page by Latex.
# Originality and significance
The paper introduces model variations that the authors claim improve performance on this particular program synthesis problem. In particular, in the DRNN decoder the hidden state is never reset for the "horizontal" (breadth-first order) decoder, and each node is only allowed to attend over the latent representation of the program. The authors claim this means their model "significantly diverges from [other] works", which seems hyperbolical.
The main contribution of this work is then the performance on the program synthesis task of Polosukhin and Skidanov. However, the model fails to improve on the Seq2Tree-guided search approach, so its main claimed benefit is that it is trained end-to-end. Although there is a strong trend in ML research to prefer end-to-end systems, it is worthwhile to ask when and why end-to-end systems are preferred. It is often clear that they are better than having separately learned components that are combined later. However, this does not apply to the model from Polosukhin et al., which consists of a single learned model being used in a search, which is a perfectly acceptable technique. In comparison, translations in neural machine translation are also produced by performing a beam search, guided by the probabilities under the model.
The paper would be significantly stronger if it could show that some alternative/existing method (e.g., a standard DRNN, or a Seq2Tree model with the same number of parameters, or a non-pretrained network) would fail to solve the problem where the authors' proposed method does not. However, the single comparison with the Seq2Tree model does not show this.
# Summary
Pros:
* Reasonable model, extensive results reported
* Decently written
Cons:
* Unclear how the performance compares to other models
* Not well justified why end-to-end methods would be better than guided-search based methods
* Model architectural differences seem relatively minor compared to original DRNN
* The pretraining using an autoencoder and the use of pretrained word embeddings seems arbitrary and is not critically evaluated
* Lack of coherent story to several results (the autoencoder performance, robustness analysis)