Neural sequence-to-sequence models are finding increasing use in editing of documents, for example in correcting a text document or repairing source code. In this paper, we argue that existing seq2seq models (with a facility to copy single tokens) are not a natural fit for such tasks, as they have to explicitly copy each unchanged token. We present an extension of seq2seq models capable of copying entire spans of the input to the output in one step, greatly reducing the number of decisions required during inference. This extension means that there are now many ways of generating the same output, which we handle by deriving a new objective for training and a variation of beam search for inference that explicitly handle this problem.
In our experiments on a range of editing tasks of natural language and source code, we show that our new model consistently outperforms simpler baselines.
This paper study the problem of editing sequences, such as natural language or code source, by copying large spans of the original sequence. A simple baseline solution to this problem is to learn a sequence to sequence neural network, which generates the edited sequence conditioned on the original one. This method can be improved by adding a copying mechanism, based on pointer networks, to copy tokens from the input. However, most of such existing approaches can only copy one input token at a time, which is a limitation when most of the input should be copied, which is the case for most editing tasks. In this paper, the authors propose a mechanism that can copy entire spans of the input instead of just individual tokens. In that case, a particular sequence can often be generated by many different actions (eg. copying individual tokens, pairs of tokens, or the whole span). It is thus important to marginalize over all the actions that generated a particular sequence. This can be done efficiently, using dynamic programming, if the probability of an action depends on the generated tokens only, but not on the sequence of actions used to generate them. In the case of neural network, this means that the decoder of the model takes the tokens as input, instead of the spans. To represent spans, the authors propose to use the concatenation of the hidden states corresponding to the beginning and end of the span. Then the probability of copying a span is obtained by taking the dot product between this representation and the current hidden state of the decoder, and applying the softmax. The authors evaluate the proposed approach on the following tasks: code repair, grammar error correction and edit representations (on wikipedia and c# code).
The paper is well written, and easy to follow, even if some sections could be a bit more detailed (for example, the section on 
beam search decoding). The problem studied in the paper - copying spans from the input - is interesting, and has applications in NLP or code generation. I think that the the proposed solution is technically sound.
However, I have some concerns regarding the paper. First I believe that many relevant prior works are not discussed in the paper, making some technical contributions of the paper not novel. For example, previous methods were proposed to copy 
spans from the input [1], to edit existing sequences [2], or to marginalize over different generation sequences
by conditioning only on the generated tokens (instead of the actions the generated the sequence) [3,4]. The body of work on iterative refinement for sequence generation is also probably relevant to this paper [5,6]. Additionally, I found the experimental section a bit weak, as most of the baseline used to compare seem a bit weak. The proposed method is mostly compared on datasets that are relatively new, or on tasks such as the grammar error correction where strong methods were excluded.
Overall, I found the paper well	written, and the proposed method to make sense. Unfortunately, I believe that the work is a bit incremental, most of the technical contributions having already been published. Since the experimental results are not very strong, I do not think the paper is good enough for publication to the ICLR conference.
== References ==
[1] Sequential Copying Networks, Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou, AAAI 2018.
[2] QuickEdit: Editing text & translations via simple delete actions, David Grangier, Michael Auli, 2017
[3] Latent Predictor Networks for Code Generation, Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Andrew Senior, Fumin Wang, Phil Blunsom, ACL 2016
[4] Training Hybrid Language Models by Marginalizing over Segmentations, Edouard Grave, Sainbayar Sukhbaatar, Piotr Bojanowski, Armand Joulin, ACL 2019
[5] Mask-Predict: Parallel Decoding of Conditional Masked Language Models, Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer, EMNLP 2019
[6] Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement, EMNLP 2018