These days deep neural networks are ubiquitously used in a wide range of tasks, from image classification and machine translation to face identification and selfdriving cars. In many applications, a single model error can lead to devastating financial, reputational and even life-threatening consequences. Therefore, it is crucially important to correct model mistakes quickly as they appear. In this work, we investigate the problem of neural network editing -how one can efficiently patch a mistake of the model on a particular sample, without influencing the model behavior on other samples. Namely, we propose Editable Training, a model-agnostic training technique that encourages fast editing of the trained model. We empirically demonstrate the effectiveness of this method on large-scale image classification and machine translation tasks.
-----------
Update after the authors' response: the authors addressed some of my concerns and presented some new results that improve the paper. I am therefore upgrading my score to "Weak Accept".
-----------
This paper proposes a way to effectively "patch" and edit a pre-trained neural network's predictions on problematic data points (e.g. where mistakes on these data points can lead to serious repercussions), without necessarily re-training the network on the entire training set plus the problematic samples. More concretely, the edit operation on the problematic samples are done using a few steps of stochastic gradient descent. 
Furthermore, the paper modifies the training procedure to make it easier for the model to perform effective patching, based on three criteria: (i) reliability (i.e. obeying the constraints specified in the edit), (ii) locality (i.e. only minimally changing the model parameters to account for the problematic samples, while ensuring the model's performance on unrelated samples remains consistent and does not degrade), and (iii) efficiency in terms of runtime and memory. To this end, the paper uses a loss term that incorporates these criteria, as weighted by interpolation coefficient hyper-parameters. The edit operation is differentiable with respect to the model parameters, which takes a similar form as the MAML approach and similarly necessitates computing second-order derivatives with respect to the model parameters. Experiments are done on CIFAR-10 toy experiments, large-scale image classification with adversarial examples, and machine translation.
Overall, while the idea of patching neural network predictions is really interesting, I have several concerns regarding the paper in its current form. I am therefore recommending a "Weak Reject". I have listed the pros and cons of this paper below, and look forward to the authors' response to the concerns I have raised.
Pros:
1. The paper is well-written, and clearly motivates the problem and why it is important.
2. The proposed approach is explained very clearly and is easy to understand.
3. The paper correctly identifies the use of distillation loss (Table 4) as a potential confound, and runs a distillation baseline without editable training as an additional baseline. This improves the thoroughness of the experiments, and ensures that the gains can really be attributed to the proposed editable training objective.
4. The paper makes interesting connections to other problems where the editable training approach can potentially be useful, such as catastrophic forgetting and adversarial training.
Cons:
1. I still have my doubts about the locality constraint. If the model makes a mistake on some problematic samples, what we want is not just fixing the model's predictions only on these problematic samples, but also on other samples that share a substantial degree of similarity to the problematic samples, so that the model can avoid making the same broader type of mistakes in the future. On the surface, the locality constraint seems to do the opposite, since it confines the impact of the edit operation only to the problematic samples. In contrast, the alternative of re-training the neural network on the full training set augmented with the problematic samples can potentially overcome this problem (since the entire model parameters are updated, correcting the model's mistakes on the problematic samples would help the model avoid making the same mistakes on other similar samples), although of course computationally much more expensive to do.
2. Related to Point 1 above, the equation defining the locality constraint (Eq. 4) seems to require an expectation of x that is drawn from p(x). How is this quantity computed? Does the model assume that x comes from the empirical distribution? Also, the "control set" (bottom of page 2) is not well defined. This is very important, since the control set defines what examples should (or should not) be changed by the editing operations. These two points should be clarified further.
3. Related to Points 1 and 2 above, I am not sure whether lower "Drawdown" is an indication of better editable training. As mentioned before, ideally the model should not only "patch" its mistakes on problematic samples, but also on samples that are substantially similar to these problematic samples. Naturally this might result in more differences from the initial model, which also means less locality (i.e. higher locality/lower drawdown may not necessarily be a good evaluation metric to evaluate edit effectiveness).   
4. The paper contains a substantial set of experiments and analysis based on toy experiments on CIFAR-10, where the labels of some examples are randomly swapped (i.e. random noise). I think this setup is dangerous, since the paper is drawing conclusions based on how well editable training can fit random noise! Instead, all the experiments should be done on real use cases (e.g. adversarial training or catastrophic forgetting). It is unfortunate that the paper has more content for the toy experiments (2.5 pages) than for the adversarial example experiment in image recognition (1.5 pages) or machine translation experiment (0.5 page).
5. The hyper-parameter experiments for selecting the learning rates can be put in the Appendix instead, rather than taking up nearly half a page in the main text (e.g. Table 1). This would leave more space for experiments, more explanation, intuition, analysis, etc.
6. The improvements (in terms of both image classification error rates or machine translation BLEU score) afforded by editable training are quite small. For instance, in the machine translation experiment, Table 6 indicates that editable training only leads to a 0.04 BLEU improvement. This is really small by standard machine translation literature, and can very well be explained by randomness in hyper-parameter initialisation, etc., rather than a better training procedure. It is hard to draw any conclusions based on such small gains.
More minor points:
1. The use of bold in the Tables are not very consistent. For instance, in Table 3, for column "Test Error Rate", 6.31% is in bold, even though baseline training has an accuracy of 6.3%, which is lower (and thus better). Also, in the "test error drawdown" column, 0.86% is in bold, even though there is a lower entry of 0.65%.
2. Missing citation to the work of Furlanello et al. (2018) for self-distillation.
References
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born Again Neural Networks. In Proc. of ICML 2018.