NEURAL PROGRAM REPAIR BY JOINTLY LEARNING TO LOCALIZE AND REPAIR
Due to its potential to improve programmer productivity and software quality, automated program repair has been an active topic of research. Newer techniques harness neural networks to learn directly from examples of buggy programs and their fixes. In this work, we consider a recently identified class of bugs called variable-misuse bugs. The state-of-the-art solution for variable misuse enumerates potential fixes for all possible bug locations in a program, before selecting the best prediction. We show that it is beneficial to train a model that jointly and directly localizes and repairs variable-misuse bugs. We present multi-headed pointer networks for this purpose, with one head each for localization and repair. The experimental results show that the joint model significantly outperforms an enumerative solution that uses a pointer based model for repair alone.
This paper presents an LSTM-based model for bug detection and repair of a particular type of bug called VarMisuse, which occurs at a point in a program where the wrong identifier is used. This problem is introduced in the Allamanis et al. paper. The authors of the paper under review demonstrate significant improvements compared to the Allamanis et al. approach on several datasets.
I have concerns with respect to the evaluation, the relation of the paper compared to the state-of-the-art in automatic program repair (APR), and the problem definition with respect to live-variable analysis.
My largest concern about both this paper and the Allamanis et al. paper is how it compares to the state-of-the-art in APR in general. There is a large and growing amount of work in APR as shown in the following papers:
[1] L. Gazzola, D. Micucci, and L. Mariani, “Automatic Software Repair: A Survey,” IEEE Transactions on Software Engineering, pp. 1–1, 2017.
[2] M. Monperrus, “Automatic Software Repair: A Bibliography,” ACM Comput. Surv., vol. 51, no. 1, pp. 17:1–17:24, Jan. 2018.
[3] M. Motwani, S. Sankaranarayanan, R. Just, and Y. Brun, “Do automated program repair techniques repair hard and important bugs?,” Empir Software Eng, pp. 1–47, Nov. 2017.
Although the proposed LSTM-based approach for VarMisuse is interesting, it seems to be quite a small delta compared to the larger APR research space. Furthermore, the above papers on APR are not referenced.
The paper under review mostly uses synthetic bugs. However, they do have a dataset from an anonymous industrial setting that they claim is realistic. In such a setting, I would simply have to trust the blinded reviewers. However, the one industrial software project tells me little about the proposed approach’s effectiveness when applied to a significant number of widely-used software programs like the ones residing in state-of-the-art benchmarks for APR, of which there are at least the following two datasets:
[4] C. L. Goues et al., “The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs,” IEEE Transactions on Software Engineering, vol. 41, no. 12, pp. 1236–1256, Dec. 2015.
[5] R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs,” in Proceedings of the 2014 International Symposium on Software Testing and Analysis, New York, NY, USA, 2014, pp. 437–440.
The above datasets are not used or referenced by the paper under review.
My final concern about the paper is the formulation of live variables. A variable is live at certain program points (e.g., program statements, lines, or tokens as called in this paper). For example, from Figure 1 in the paper under review, at line 5 in (a) and (b), object_name and subject_name are live, not just sources.  In the problem definition, the authors say that "V_def^f \subseteq V denotes the set of all live variables", which does not account for the fact that different variables are alive (or dead) at different points of a program. The authors then say that, for the example in Figure 1, "V_def^f contains all locations in the program where the tokens in V appear (i.e., tokens in the Blue boxes), as well as token sources from line 1”. The explanation of the problem definition when applied to the example does not account for the fact that different variables are alive at different program points. I’m not sure to what extent this error negatively affects the implementation of the proposed model. However, the error could be potentially quite problematic.