EXECUTION-GUIDED NEURAL PROGRAM SYNTHESIS
Neural program synthesis from input-output examples has attracted an increasing interest from both the machine learning and the programming language community. Most existing neural program synthesis approaches employ an encoder-decoder architecture, which uses an encoder to compute the embedding of the given inputoutput examples, as well as a decoder to generate the program from the embedding following a given syntax. Although such approaches achieve a reasonable performance on simple tasks such as FlashFill, on more complex tasks such as Karel, the state-of-the-art approach can only achieve an accuracy of around 77%. We observe that the main drawback of existing approaches is that the semantic information is greatly under-utilized. In this work, we propose two simple yet principled techniques to better leverage the semantic information, which are execution-guided synthesis and synthesizer ensemble. These techniques are general enough to be combined with any existing encoder-decoder-style neural program synthesizer. Applying our techniques to the Karel dataset, we can boost the accuracy from around 77% to more than 90%.
This paper proposes guiding program synthesis with information from partial/incomplete program execution. The idea is that by executing partial programs, synthesizers can obtain the information of the state the (partial) program ended in and can, therefore, condition the next step on that (intermediate) state. The paper also mentions ensembling synthesizers to achieve a higher score, and by doing that it outperforms the current state-of-the-art on the Karel dataset program synthesis task.
In general, I like the idea of guiding synthesis with intermediate executions, and the evaluation in the paper shows this does make sense, and it outperforms the SOTA. The idea is original and the evaluation shows it is significant (enough). However, I have two major concerns with the paper, its presented contribution, and the clarity.
First, I cannot accept ensembling as a contribution to this paper. There is nothing novel about the ensemble proposed, and ensembling, as a standard method that pushes models that extra few percentage points, is present in a lot of other research. I have nothing against achieving SOTA results with it, while at the same time showing that the best performing model outperforms previous SOTA, which this paper orderly does. However, I cannot accept non-novel ensembling as a contribution of the paper.
Second, the clarity of the paper should be substantially improved:
- my main issue is that it is not clear how the Exec algorithm (see next point too) is trained. From what I understand Exec is trained on supervised data via MLE. What is the supervised data here?  Given the generality claims and the formulation in Algorithm 1/2, and possible ways one could use the execution information, as well as the fact that the model should be end-to-end trainable via MLE, it seems to me that the model is trained on prefixes (defined by Algorithm 1/2) of programs. Whether this is correct or not, please provide full details on how one can train Exec without using RL.
- By looking at Table 3, it seems that the generalization boost coming from Exec (I’m ignoring ensembling) is higher enough, and that’s great. However, it’s obvious that the exact match gain by Exec is minute, implying that the proposed algorithm albeit great on the generalization metric, does not improve the exact match at all. Do you have any idea why is that? Is that because Exec is trained via MLE and the Exec algorithm doesn’t add anything new to the training procedure?
- how do algorithm 1 and 2 exactly relate? I guess there is a meaning of ellipses in Lines 1 and 13, however, that is not mentioned anywhere. Is the mixture of algorithm 1 and 2 (and a non-presented algorithm for while loops) the Exec algorithm? How exactly are these algorithms joined, i.e what is the final algorithm?
- while on one side, I find some formalizations (problem definitions, definition 1, semantic rules in table 2) nicely done, I do not see their necessity nor big gains from them. In my opinion, the understanding of the rest of the paper does not depend on them, and they are well-described in the text.
- the paper says that the algorithm “helps boost the performance of different existing training algorithms”, however, it does so only on the Bunel et al model (and the MLE baseline in it), and albeit there’s mention of the generality, it has not been shown on anything other than those two models and the Karel dataset.
- do lines 6-7 in Algorithm 2 recurse? Does the model support arbitrarily nested loops/if statements?
- The claim that the shortest principle is most effective is supported by 2 data points, without any information on the variance of the prediction/dependence on the seed. Did you observe this for #models > 10 too? Up to what number?
- In table 3, is Exec on MLE? Could you please, for completeness, present the results of Exec + RL + ensemble in the table too?
- summarization, point 3 - what are the different modules mentioned here? Exec/RL/ensemble?
Minor issues, remarks, typos:
- table 1 position is very unfortunate
- figure 1 is not self-explanatory - it takes quite a lot of space to explain the network architecture, yet it fails to deliver meaning to parts of it (e.g. what is h_t^x, why is it max-pooled, what is g_t, etc)
- abstract & introduction - “Reducing error rate around 60%” absolute percentage points seem like a better evaluation measure (that the paper does use). Why is the error rate reduction necessary here?
- figure 2 - why is the marker in one of the corners, and not in the cell itself?
- Algorithm 1, step 4, is this here just as initialization, so S is non-empty to start with?
- Table 2 rule names are unclear (e.g. S-Seq-Bot ?)
- Table 3 mentions what Exec indicates twice