Learning to Represent Edits
We introduce the problem of learning distributed representations of edits. By combining a "neural editor" with an "edit encoder", our models learn to represent the salient information of an edit and can be used to apply edits to new inputs. We experiment on natural language and source code edit data. Our evaluation yields promising results that suggest that our neural network models learn to capture the structure and semantics of edits. We hope that this interesting task and data source will inspire other researchers to work further on this problem.
 This paper looks at learning to represent edits for text revisions and code changes. The main contributions are as follows: * They define a new task of representing and predicting textual and code changes * They make available a new dataset of code changes (text edit dataset was already available) with labels of the type of change * They try simple neural network models that show good performance in representing and predicting the changes The NLP community has recently defined the problem of predicting atomic edits for text data (Faraqui, et al. EMNLP 2018, cited in the paper), and that is the source of their Wikipedia revision dataset. Although it is an interesting problem, it is not immediately clear from the Introduction of this paper what would be enabled by accurate prediction of atomic edits (i.e. simple insertions and deletions), and I hope the next version would elaborate on the motivation and significance for this new task. The "Fixer" dataset that they created is interesting. Those edits supposedly make the code better, so modeling those edits could lead to "better" code. Having that as labeled data enables a clean and convincing evaluation task of predicting similar edits. The paper focuses on the novelty of the task and the dataset, so the models are simple variations of the existing bidirectional LSTM and the gated graph neural network. Because much of the input text (or code) does not change, the decoder gets to directly copy parts of the input. For code data, the AST is used instead of flat text of the code. These small changes seem reasonable and work well for this problem. Evaluation is not easy for this task. For the task of representing the edits, they show visualizations of the clusters of similar edits and conduct a human evaluation to see how similar these edits actually are. This human evaluation is not described in detail, as they do not say how many people rated the similarity, who they were (how they were recruited), how they were instructed, and what the inter-rater agreement was. The edit prediction evaluation is done well, but it is not clear what it means when they say better prediction performance does not necessarily mean it generalizes better. That may be true, but then without another metric for better generalization, one cannot say that better performance means worse generalization. Despite these minor issues, the paper contributes significantly novel task, dataset, and results. I believe it will lead to interesting future research in representing text and code changes.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning to Represent Edits
We introduce the problem of learning distributed representations of edits. By combining a "neural editor" with an "edit encoder", our models learn to represent the salient information of an edit and can be used to apply edits to new inputs. We experiment on natural language and source code edit data. Our evaluation yields promising results that suggest that our neural network models learn to capture the structure and semantics of edits. We hope that this interesting task and data source will inspire other researchers to work further on this problem.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper looks at learning to represent edits for text revisions and code changes. The main contributions are as follows: * They define a new task of representing and predicting textual and code changes * They make available a new dataset of code changes (text edit dataset was already available) with labels of the type of change * They try simple neural network models that show good performance in representing and predicting the changes The NLP community has recently defined the problem of predicting atomic edits for text data (Faraqui, et al. EMNLP 2018, cited in the paper), and that is the source of their Wikipedia revision dataset. Although it is an interesting problem, it is not immediately clear from the Introduction of this paper what would be enabled by accurate prediction of atomic edits (i.e. simple insertions and deletions), and I hope the next version would elaborate on the motivation and significance for this new task. The "Fixer" dataset that they created is interesting. Those edits supposedly make the code better, so modeling those edits could lead to "better" code. Having that as labeled data enables a clean and convincing evaluation task of predicting similar edits. The paper focuses on the novelty of the task and the dataset, so the models are simple variations of the existing bidirectional LSTM and the gated graph neural network. Because much of the input text (or code) does not change, the decoder gets to directly copy parts of the input. For code data, the AST is used instead of flat text of the code. These small changes seem reasonable and work well for this problem. Evaluation is not easy for this task. For the task of representing the edits, they show visualizations of the clusters of similar edits and conduct a human evaluation to see how similar these edits actually are. This human evaluation is not described in detail, as they do not say how many people rated the similarity, who they were (how they were recruited), how they were instructed, and what the inter-rater agreement was. The edit prediction evaluation is done well, but it is not clear what it means when they say better prediction performance does not necessarily mean it generalizes better. That may be true, but then without another metric for better generalization, one cannot say that better performance means worse generalization. Despite these minor issues, the paper contributes significantly novel task, dataset, and results. I believe it will lead to interesting future research in representing text and code changes.
assistant
 This paper looks at learning to represent edits for text revisions and code changes. The main contributions are as follows: * They define a new task of representing and predicting textual and code changes * They make available a new dataset of code changes (text edit dataset was already available) with labels of the type of change * They try simple neural network models that show good performance in representing and predicting the changes The NLP community has recently defined the problem of predicting atomic edits for text data (Faraqui, et al. EMNLP 2018, cited in the paper), and that is the source of their Wikipedia revision dataset. Although it is an interesting problem, it is not immediately clear from the Introduction of this paper what would be enabled by accurate prediction of atomic edits (i.e. simple insertions and deletions), and I hope the next version would elaborate on the motivation and significance for this new task. The "Fixer" dataset that they created is interesting. Those edits supposedly make the code better, so modeling those edits could lead to "better" code. Having that as labeled data enables a clean and convincing evaluation task of predicting similar edits. The paper focuses on the novelty of the task and the dataset, so the models are simple variations of the existing bidirectional LSTM and the gated graph neural network. Because much of the input text (or code) does not change, the decoder gets to directly copy parts of the input. For code data, the AST is used instead of flat text of the code. These small changes seem reasonable and work well for this problem. Evaluation is not easy for this task. For the task of representing the edits, they show visualizations of the clusters of similar edits and conduct a human evaluation to see how similar these edits actually are. This human evaluation is not described in detail, as they do not say how many people rated the similarity, who they were (how they were recruited), how they were instructed, and what the inter-rater agreement was. The edit prediction evaluation is done well, but it is not clear what it means when they say better prediction performance does not necessarily mean it generalizes better. That may be true, but then without another metric for better generalization, one cannot say that better performance means worse generalization. Despite these minor issues, the paper contributes significantly novel task, dataset, and results. I believe it will lead to interesting future research in representing text and code changes.