A Retrieve-and-Edit Framework for Predicting Structured Outputs
For the task of generating complex outputs such as source code, editing existing outputs can be easier than generating complex outputs from scratch. With this motivation, we propose an approach that first retrieves a training example based on the input (e.g., natural language description) and then edits it to the desired output (e.g., code). Our contribution is a computationally efficient method for learning a retrieval model that embeds the input in a task-dependent way without relying on a hand-crafted metric or incurring the expense of jointly training the retriever with the editor. Our retrieve-and-edit framework can be applied on top of any base model. We show that on a new autocomplete task for GitHub Python code and the Hearthstone cards benchmark, retrieve-and-edit significantly boosts the performance of a vanilla sequence-to-sequence model on both tasks.
This paper addresses the task of highly structured output prediction in the contexts of, here, source code generation.  The authors propose a two-step process for this, consisting of a retrieval model that retrieves the closest seen training instance to the test instance, and an edit model, that edits the retrieved training instance.  Quality: - The authors approach the relatively recently proposed task of generating Python source code. Unlike others, they work in a setting with less supervision, where they assume no access to the Abstract Syntax Tree.  - The key innovation here is the task-specific retrieval model, trained using a variational autoencoder that is then further fine-tuned such that the output distribution of the learned encoder resembles the oracle encoder. I like this idea and the proposed model a lot and can see how it would be useful beyond the immediate scope of the paper. - It is clear how the proposed method differs from previous work, and related work is adequately cited  - What wasn't clear to was why the retriever only retrieves one training example to be edited per testing instance. This seems to rely very heavily on the retriever's performance and also on the training set containing one perfectly related training instance. Wouldn't it be better to retrieve several and then try and then have the edit model merge them as well? - The authors could have shown more ablation studies for their results. They make several assumptions in their approach for which detailed results are not shown, e.g. the encoding prior (Section 3.1.1) - They assume that there is an oracle editor, but don't describe how the oracle decoder is constructed from the training data. Also, they didn't discuss the dependence of their approach on the oracle editor or how could train the model in a scenario without access to an oracle editor.   Clarity: - The submission is overall well-written and well-motivated. The figures provided nicely illustrate the approach. - In several places in the paper, abbreviations are used without being introduced (e.g. "AST" in abstract) and variables are introduced after formulae they are used in, particularly in Section 3. Small rewrites could make the paper more readable in those places.  Originality: - This work is a novel combination of existing techniques, combining variational autoencoders, joint training, and sequence models. Previous work using retrieval as part of their approach saw this mainly as a black-box component and did not try to jointly train the retrieval component with the main model.  Significance: - As already stated above, I believe the general idea as well as the model proposed here will be used beyond the immediate scope of this work (Python code generation).  =================  Thanks to the authors for clarifying my questions. My remaining concerns are addressed and I would be happy to see this paper appear at NIPS. In light of this, I increased my score by 1.