Models of code can learn distributed representations of a program's syntax and semantics to predict many non-trivial properties of a program. Recent state-ofthe-art models leverage highly structured representations of programs, such as trees, graphs and paths therein (e.g., data-flow relations), which are precise and abundantly available for code. This provides a strong inductive bias towards semantically meaningful relations, yielding more generalizable representations than classical sequence-based models. Unfortunately, these models primarily rely on graph-based message passing to represent relations in code, which makes them de facto local due to the high cost of message-passing steps, quite in contrast to modern, global sequence-based models, such as the Transformer. In this work, we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias: Graph Sandwiches, which wrap traditional (gated) graph message-passing layers in sequential message-passing layers; and Graph Relational Embedding Attention Transformers (GREAT for short), which bias traditional Transformers with relational information from graph edge types. By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation. Starting with a graph-based model that already improves upon the prior state-of-the-art for this task by 20%, we show that our proposed hybrid models improve an additional 10-15%, while training both faster and using fewer parameters.
The paper proposes improvements on existing probabilistic models for code that predicts and repairs variable misuses. This is a variant of the task, proposed by Vasic et al. The task takes a dataset of python functions, introduces errors in these functions and makes a classifier that would identify what errors were introduced and effectively reconstruct the original code.
The paper claims to improve state-of-the-art results published by Vasic et al for this task, however the RNN model by Vasic et al was known to be far from optimal when the work was published. Furthermore, that task was evaluated on artificially introduced changes (the original code could contain an error), but it is not clear that the improvements would have any practical effect. In fact, I conjecture that the bug-detector is in fact worse, because the entire dataset is not sufficiently large for millions of parameters and it is not clear that bugs that were originally the dataset were not learned by the better model, making it worse at spotting them. Given the relatively thinner contribution on the rest of the paper, I think this would be a valid question to be addressed to show the effectiveness of the model beyond accuracy on the artificial task.
The paper does a number of contributions to the neural architecture. The most important change precision-wise is to use transformer model instead of RNN (the model used by Vasic et al).  This change is also what makes the work perform as well or better than GGNN-based approaches. The paper then proposes to improve the transformer model by modifying the attention where there are edges. The rest of the contributions seem to be addressing the problem of aster convergence speed.  The other contribution of the paper is by selecting which edges to include and it is also shown to improve convergence speed.
Given that most of the work talks about performance, it would also help if the authors clarify what kind of hardware was used and which optimizer.
Q: Why a larger transformer model was not evaluated?
More minor issues:
“We conjecture that the Transformer learns to infer many of the same connections”. There is no confirmation for this besides similar accuracy, but if this is the case, why would I change the architecture and not just try with initializing the vectors to values corresponding to this knowledge and get faster convergence?
page 3, “where q and k correspond to the query and key vectors as described above,”. It seems it is q_i and k_j?
Update after the rebuttal:
 - I thank the authors for running additional experiments on a short notice. I have some reservations about their correctness though (we still do not know if there were bugs fixed in these commits, their number is very low and the authors seem to have cherrypicked the numbers to show - their first updated revision had recall at 20%, then decided to show it at 10%, the RNN baseline is actually having the highest recall of all although it is not highlighted). I am not sure that data cleaning of this small evaluation sample will not show a different picture.
 - I actually increase my score a bit (I was torn in the beginning), because this is one of the first papers to run transformer model on code. I still think the actual contributions of the paper are minor.