Programming languages are emerging as a challenging and interesting domain for machine learning. A core task, which has received significant attention in recent years, is building generative models of source code. However, to our knowledge, previous generative models have always been framed in terms of generating static snapshots of code. In this work, we instead treat source code as a dynamic object and tackle the problem of modeling the edits that software developers make to source code files. This requires extracting intent from previous edits and leveraging it to generate subsequent edits. We develop several neural networks and use synthetic data to test their ability to learn challenging edit patterns that require strong generalization. We then collect and train our models on a large-scale dataset consisting of millions of fine-grained edits from thousands of Python developers. From the modeling perspective, our main conclusion is that a new composition of attentional and pointer network components provides the best overall performance and scalability. From the application perspective, our results provide preliminary evidence of the feasibility of developing tools that learn to predict future edits.
Summary: The authors study building models for edits in source code. The application is obvious: a system to accurately predict what the next edit should be would be very valuable for developers. Here, edits are modeled by two types of sequences: one that tracks the state of all edits at each time step (and is thus very long), and one that contains the initial step and a changelist that contains the minimal information required to derive the state at any time. The authors train models on top of both of these representations, with the idea being to match the performance of the explicit (heavy) model with the implicit model. This is shown to be challenging, but a clever model is introduced that achieves this, and is thus the best of both worlds. There are synthetic and real-world code (text) edit experiments.
Strengths: The problem is well-posed and well-motivated. There's a nice application of powerful existing models, combined and tailored to the current work. The writing is generally quite clear. The number of experiments is quite solid. 
Weaknesses: The main flaw is that nothing here is really specifically for souce code; the authors are really just modeling edits in text sequences. There's not an obvious way to integrate the kinds of constraints that source code typically satisfies either. There's some confusion (for me) about the implicit/explicit representations. More questions below.
Verdict: This is a pretty solid paper. It doesn't quite match up to its title, but it sets out a clearly defined problem, achieves its aims, and introduces some nice tricks. Although it doesn't produce anything genuinely groundbreaking, it seems like a nice step forward.
Comments and Questions:
- The problem is written in the context of source code, but it's really setup just for text sequences, which is a broader problem. Is there a way the authors take can advantage of the structural requirements for source code? I don't see an obvious way, but I'm curious what the authors think.
- What's the benefit of using the implicit representation for the positions? The explicit/implicit  position forms are basically just using the permutation or the inverse permutation form, which are equivalent. I don't see directly what's saved here, the alphabet size and the number of integers to store is the same.
- Similar question. The implicit likelihood is s^0, e^(1),...,e^(t-1), with the e^(i)'s being based on the implicit representations of the positions. Seems like you could do this with the *explicit* positions just fine, they carry enough information to derive s^(i) from s^(i-1). That is, the explicit/implicit problems are not really related to the explicit/implicit position representations.
- Just wanted to point out that this type of approach to sequences and edits has been studied pretty often in the information/coding theory communities, especially in the area of synchronization. There, the idea is to create the minimal "changelist" of insertions/deletions from two versions of a file. This could come in handy when building the datasets. See, for example, Sala et al "Synchronizing files from a large number of insertions and deletions".
- The problem statement should be stated a bit more rigorously. We'd like to say that the initial state is drawn from some distribution and that the state at each time forms a stochastic process with some transition law. As it stands the problem isn't well-defined, since with no probability distribution, there's nothing to predict and no likelihood.
- The "analogical decoder" idea is really nice.
- For the synthetic dataset, why are you selecting a random initial string, rather than using some existing generative text or source code model, which would get you synthetic data that more closely resembles code?
- I really liked the idea of using an oracle that gives the position as upper bound. Would it make sense to also have the opposite oracle that gives the edit symbol, but doesn't tell the location? I'm really curious which is the "harder" task, predicting the next symbol or the next location. In the information-theory setting, these two are actually equally hard, but the real-world setting might be pretty different. It would also be interesting to train models on top of the POMP. That would produce genuine upper bounds to the model performances. 
- The explicit baseline model performs very well on all the edit types in Table 1. Are there cases where even this explicit case works poorly? Is the improved implicit model *always* upper bounded by the explicit model (to me it seems like the answer should always be yes, but it would be interesting to check it out for cases where explicit is not very high accuracy).