We present a learning-based approach to detect and fix a broad range of bugs in Javascript programs. We frame the problem in terms of learning a sequence of graph transformations: given a buggy program modeled by a graph structure, our model makes a sequence of predictions including the position of bug nodes and corresponding graph edits to produce a fix. Unlike previous works that use deep neural networks, our approach targets bugs that are more complex and semantic in nature (i.e. bugs that require adding or deleting statements to fix). We have realized our approach in a tool called HOPPITY. By training on 290,715 Javascript code change commits on Github, HOPPITY correctly detects and fixes bugs in 9,490 out of 36,361 programs in an end-to-end fashion. Given the bug location and type of the fix, HOPPITY also outperforms the baseline approach by a wide margin.
This paper proposes a graph tranformation-based code repair tool. By representing source code as a graph a network is asked to take a series of simple graph edit operations to edit the code. The authors show that their method better predicts edits from existing code.
Overall, I find the problem interesting and the neural approach of the authors reasonable, principled and interesting. However, the evaluation is either badly written or the authors have not understood the related literature they are comparing to. It is thus unclear how this model compares to alternatives. I believe that this is good work that needs to eventually be published, but it's not ready at this time.
* In 3.2.1 "Value" paragraph "Instead of predicting the replacement value using a language generative model (Chen et al 2018, Allamanis et al 2018), we let the model to choose from either the values appearing in the current file." 
  
   -The Chen et al. (2018) paper is indeed a translation-style model, but thanks to the copying mechanism, the model can also choose from values appearing in the current input.
   - The Allamanis et al. (2018) is *not* a generative model and it can only select a misused variable for those that exist on within the scope (for the Variable Misuse Task)
* The authors compare to the VARNAMING (Section 6.2) task of Allamanis et al. (2018). This is odd, since to my understanding, this task is about alpha-renaming, i.e. semantics preserving renaming of all identifiers of a variable. This is not comparable with the REP_VAL action which simply replaces the value of a single leaf node from the value table. The VARMISUSE objective (picking one variable directly from the value table) is the closest analogue, although it performs no localization. 
Thus, it is not surprising that a char-level language model cannot predict the names of the elements in the value table. So, I don't think that we learn anything from the current comparison.
* It's unclear to me why the REP_TYPE action is relevant to VARMISUSE task of Allamanis et al. (2018). This seems to simply be a classification task (among all known/valid types of the tree) given an existing position.
* Given the above two points, it's unclear what the comparison with the baseline (Table 3) means and what one can deduce from the evaluation: The Allamanis et al. (2018) work tries to find the correct variable at a given location; this work tries to pick a location and find a different value from current one. The Allamanis et al. (2018) and Cvitkovic et al. (2018) uses additional semantic information (as edges), importantly data flow and control flow. This work uses none of those. I am not sure that I can see how the comparison here can work.
* In 3.1 "Unlike previous approaches, we additionally add value nodes that store the actual content of the leaf nodes [...]. The purpose of introducing this additional set of nodes is to provide a name-independent strategy for code representation and modification". I am not sure what's unlike previous approaches. For example, the Allamanis et al. (2018) and the Cvitkovic et al. (2018) work, connect the same variables with various ways (e.g. data-flow edges) in a way that provides a name-independent strategy for code representation. I would fing it surprising that a principled comparison between Allamanis et al. (2018)/ Cvitkovic et al. (2018) and this work will yield any improvements: all papers use the same model (variations of a graph neural network) and this work uses strictly fewer information (no data flow, control flow edges). A comparison would simply compare how the two graph-extraction methods compare, but nothing valuable otherwise.
* The authors of this work frequently refer to Chen et al. (2018), but never compare with it. I would expect that this model beats the Chen et al. (2018) work, but this needs to be shown. In my opinion this is a much more relevant baseline, compared to the comparison with Allamanis et al. (2018). If the authors insist on comparing with that work, then a "fair" comparison needs to be made. Comparing between REP_VAL and VARMISUSE given the location of the node-to-be-repaired, is probably the only reasonable option. Comparing with Vasic et al. (2019) for the REP_VAL/REP_TYPE would also be reasonable.
* The correctness of the NO_OP predictions are never evaluated. Does the model "know" when to terminate or does it keep editing? It's unclear what is the overall accuracy (for a full edit).
Other questions:
* The "ADD" operation predicts two locations: one for the parent and one for the left sibling of the node. Does this mean that for a given node the "ADD" operation will never be able to add a left-most child (since it has no left sibling)?
* It has been recently found that code corpora collected by scraping GitHub may contain a disproportionate amount of duplicates (Lopes et al. 2017, Allamanis 2018). It is unclear if the authors have taken any steps to detect and remove duplicates that would affect their results.
* In Table 2: Does "Beam-3" mean "accuracy in the top 3" or does it mean "accuracy of the top prediction when the beam size is 3"? The difference between "Beam-1" and "Beam-3" is surprisingly large. Can you explain why?
* In the TAJS comparison, it is unclear if any negative examples are added: it's unclear what's the false positive ratio of HOPPITY vs TAJS.
Minor:
* Abstract: "Github" -> "GitHub"
* 3.2 REP_VAL: "Tthe"-> "The"
* Sec 7: "Vasic et al. (2019) present a pointer network..." It is unclear if this work outperforms the Allamanis et al. (2018) work as the comparison is only partial. Another ICLR submission https://openreview.net/forum?id=B1lnbRNtwr suggests that this is not the case.
* The citation of Chen et al. (2018) should be capitalized correctly "Sequencer"->"SequenceR"
## References
Allamanis, Miltiadis. "The Adverse Effects of Code Duplication in Machine Learning Models of Code." arXiv preprint arXiv:1812.06469 (2018).
Cvitkovic, Milan, Badal Singh, and Anima Anandkumar. "Open Vocabulary Learning on Source Code with a Graph-Structured Cache." arXiv preprint arXiv:1810.08305 (2018).
Lopes, Cristina V., et al. "DéjàVu: a map of code duplicates on GitHub." Proceedings of the ACM on Programming Languages 1.OOPSLA (2017): 84.