Machine learning models that take computer program source code as input typically use Natural Language Processing (NLP) techniques. However, a major challenge is that code is written using an open, rapidly changing vocabulary due to, e.g., the coinage of new variable and method names. Reasoning over such a vocabulary is not something for which most NLP methods are designed. We introduce a Graph-Structured Cache to address this problem; this cache contains a node for each new word the model encounters with edges connecting each word to its occurrences in the code. We find that combining this graph-structured cache strategy with recent Graph-Neural-Network-based models for supervised learning on code improves the models' performance on a code completion task and a variable naming task -with over 100% relative improvement on the latter -at the cost of a moderate increase in computation time.
(updated with some summaries from discussion over the initial review)
The paper discusses the topics of predicting out-of-vocabulary tokens in programs abstract syntax trees. This could have application in code completion and more concretely two tasks are evaluated:
 - predicting a missing reference to a variable (called FillInTheBlank)
 - predicting a name of a variable (NameMe)
Unfortunately, the paper proposes overly complex and strange formulations of these tasks, heavy implementation with unnecessary (non-motivated) neural architectures and as a result, does not demonstrate state-of-the-art performance or precision on comparable tasks. Figure 1 shows the complexity of the approach, with multiple steps of building a graph, introducing the vocabulary cache to then produce a vector at every node of the input tree of the program (instead of creating architecture for a given task), yet simple analysis over which variables can be chosen is missing.
The FillInTheBlank task is badly defined already on the running example. The goal is to select a variable to fill in a blank and already in the example on Figure 2, one of the candidate variables is out of scope at the location to fill. The motivation for the proposed formulation with building a graph and then computing attention over nodes in that graph is unclear and experiments do not help it. For example, [1] (also cited in the paper) solves the same problem more cleanly by considering only the variables in the scope*. There is no good experimental comparison to that work, but it is unlikely it will perform worse. Also [1] does not suffer from vocabulary problems for that task.
Summary discussion below: the experiments here are incomparable on many levels with prior works: different architecture details, different even smaller dataset than from [1]. There is a third-party claim that on a full system, the general idea improves performance, but I take it with a grain of salt as no clean experiment was yet done. The reviewer notes that the authors disagree the baselines are not meaningful.
The NameMe tasks also shows the weakness of the proposed architectures. This work proposes to compute vectors at every node where a variable occurs and then to average them and decode the variable name to predict. In comparison, several prior works introduce one node per variable (not per occurrence), essentially removing the long distance relationships between occurrences of the same variable variables and removing the need to average vectors and enforcing the same name representation at every occurrence of the variable [name]. The setup here is incomparable to specialized naming prior works, one feature (a node per variable) is replaced with another (a node per subtoken), but for baselines authors choose to only to be similar to [1]. Also, while not on the same dataset, [2,3] consistently get higher accuracy on a related and more complicated task of predicting multiple names at the same time over multiple programming languages and with much simpler linear models. This is not surprising, because they propose simpler architectures better suited for the NameMe task.
[1] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs
with graphs. ICLR 2017
[2] Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from Big
Code
[3] Uri Alon, Meital Zilberstein, Omer Levy, Eran Yahav. A General Path-Based Representation for Predicting Program
* corrected text