We address the problem of Any-Code-to-Code Generation (AnyC2C) -generating code given its surrounding code without any restriction on the vocabulary or structure. The state-of-the-art in this problem is the sequence-to-sequence (seq2seq) approach, which treats code as a sequence and does not leverage any structural information. We introduce a new approach to AnyC2C that leverages the strict syntax of programming languages to model a code snippet as a tree -structural language modeling (SLM). SLM estimates the probability of the program's abstract syntax tree (AST) by decomposing it into a product of conditional probabilities over its nodes. We present a neural model that computes these conditional probabilities by considering all AST paths leading to a target node. Unlike previous structural techniques that have severely restricted the kinds of expressions that can be generated in this task , our approach can generate arbitrary expressions in any programming language. Our model significantly outperforms both seq2seq and a variety of existing structured approaches in generating Java and C# code. We make our code, datasets, and models available online.
This paper presents a grammar-based generation approach for "slot-filling" style code generation tasks. Given a context AST with opening non-terminal node, the model completes the opening node by predicting a sequence of child nodes, which forms a sub-AST rooted at the original opening node. The proposed model encodes context ASTs using a path-based approach (Alon et al., 2019a), essentially generalizing the previous model of Alon et al., 2019a from a code-to-sequence setting (e.g., generating natural language comments from code) to a "code-to-code" setting (i.e., code completion given contextual snippets). 
Strong Points:
* The paper is very well written. The idea of formalizing code completion as structured language modeling and extending Alon et al., 2019a for the task is natural and well executed, with strong models and significantly improved results on two code completion benchmarks for both Java and C#.
* The authors attempted to establish comparisons with most existing code generation models.
Detailed Review:
*Technical Contribution* I have a very mixed feeling with this paper, while the model registers high empirical performance, the technical contribution is a bit limited, as detailed below:
    - *Path-based Context Encoding* The most important contribution in this submission is the application of path-based AST encoding model of Alon et al., 2019a to encode context (the given contextual and partially generated ASTs) for code generation. While the path-based encoding scheme is indeed a powerful model that intuitively encapsulates and generalizes over most previous approaches (Section 7), applying the model to a different task without significant task-specific adaptation or in-depth analysis might not sound technically novel. Meanwhile, the core idea of modeling/encoding the information flow in both the given context AST and partially generated programs for opening node expansion has already been explored in Brockschmidt et al. (2019a), albeit using a different encoding approach (GNNs) and in a relatively restricted setting of generating arithmetic expressions.
    - *Node-based Tree Generation Model* Apart from the path-based context encoding model, the node-based generation model presented in Section 2 also seems interesting. However, it might take longer time-steps to generate the node sequence instead of the sequence of production rules (composed of multiple child nodes), which could make optimization and inference more difficult. On the other hand, to control arity, the node-based approach need to inject synthetic "EOS" nodes to signal end of generating an argument sequence, while existing production rule-based systems could easily generate arbitrary number of argument nodes using either a transition system (e.g., Yin and Neubig, 2018) or a special neural component to compute end-of-argument-list probability (e.g., Rabinovich et al. (2017)), without using separate production rules of different arity.
    - *Syntactic Copy Mechanism* While the proposed syntactic terminal token copy mechanism (Section 3.3) could be better than the vanilla sequential one, there have already been syntactic copying models capable of copying both terminal tokens and partial ASTs from the context (Yin et al., 2019). 
    How to Improve: to better understand the different technical contributions outlined above and their relative impacts, the following ablation studies would be helpful:
        - Importance of Path-based Context Encoding: the Seq→Path ablation in Table 3 alone might not be adequate to demonstrate the importance of path-based encoding of AST contexts for code generation tasks. The authors should compare with the GNN-based context encoding approach in Brockschmidt et al. (2019a) as this is the most relevant work. The original GNN→NAG model cited in Table 2 used a much simpler copying mechanism and a vanilla production-based tree generation model, and therefore not directly comparable with a tuned SLM.
        - Importance of Node-based Tree Generation Model: If possible, the authors might consider swapping their node-based tree generation model with a state-of-the-art production-based approach (e.g., Yin et al., 2019) to demonstrate its effectiveness.
*Claims* The authors claimed in the beginning of the paper that previous program synthesis approaches are either restricted in domains (e.g., DSLs like SQL) or grammars (e.g., restricted grammar of the full language), therefore coining the proposed approach as "any-code generation". However, there are indeed code generation systems (some of them cited in this paper) that synthesize open-domain code snippets in general-porpuse programming languages without restriction on vocabulary or grammar. To give a few examples, Iyer et al. (2018) generate open-domain Java class member functions; Rabinovich et al. (2017) and Zhao et al. (2019) predict full Python classes or partial snippets, while Yin et al. (2019) synthesize open-domain short C# code diffs observed in GitHub commits. In fact, the the proposed "any-code generation" benchmark is limited to sub-expressions defined within a function, whose scope is more restricted than other benchmarks like CONCODE. 
    How to improve: the authors might present more evidence to substantiate the their claim on the novelty of the AnyGen benchmark compared with existing open-domain, general purpose code synthesis benchmarks, or consider revising the claim and the title.
References:
* Zhao et al., Neural Networks for Modeling Source Code Edits. 2019
* Yin et al., Learning to Represent Edits. 2019