mixup: BEYOND EMPIRICAL RISK MINIMIZATION
Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.
I enjoyed reading this well-written and easy-to-follow paper. The paper builds on the rather old idea of minimizing the empirical vicinal risk (Chapelle et al., 2000) instead of the empirical risk. The authors' contribution is to provide a particular instance of vicinity distribution, which amounts to linear interpolation between samples. This idea of linear interpolation on the training sample to generate additional (adversarial, in the words of the authors) data is definitely appealing to prevent overfitting and improve generalization performance at a mild computational cost (note that this comment does not just apply to deep learning). This notion is definitely of interest to machine learning, and to the ICLR community in particular. I have several comments and remarks on the concept of mixup, listed below in no particular order. My overall opinion on the paper is positive and I stand for acceptance, provided the authors answer the points below. I would especially be interested in discussing those with the authors.
1 - While data augmentation literature is well acknowledged in the paper, I would also like to see a comment on domain adaptation, which is a very closely related topic and of particular interest to the ICLR community.
2 - Paragraph after Eq. (1), starting with "Learning" and ending with "(Szegedy et al., 2014)": I am not so familiar with the term memorization, is this just a fancy way of talking about overfitting? If so, you might want to rephrase this paragraph with terms more used in the machine learning community. When you write "one trivial way to minimize [the empirical risk] is to memorize the training data", do you mean output a predictor which only delivers predictions on $X_i$, equal to $Y_i$? If so, this is again not specific to deep learning and I feel this should be a bit more discussed.
3 - I have not found in the paper a clear heuristics about how pairs of training samples should be picked to create interpolations. Picking at random is the simplest however I feel that a proximity measure on the space $\mathcal{X}$ on which samples live would come in handy. For example, sampling with a probability decreasing as the Euclidean distance seems a natural idea. In any case, I strongly feel this discussion is missing in the paper.
4 - On a related note, I would like to see a discussion on how many "adversarial" examples should be used. Since the computational overhead cost of computing one new sample is reasonable (sampling from a Beta distribution + one addition), I wonder why $m$ is not taken very large, yielding more accurate estimates of the empirical risk. A related question: under what conditions does the vicinal risk converge (in expectation for example) to the empirical risk? I think some comments would be nice.
5 - I am intrigued by the last paragraph of Section 5. What do the authors exactly have in mind when they suggest that mixup could be generalized to regression problems? As far as I understood the paper, since $\tilde{y}$ is defined as a linear interpolation between $y_i$ and $y_j$, this formulation only works for continuous $y$s, like in regression. This formulation is not straightforwardly transposable to classification for example. I therefore am quite confused about the fact that the authors present experiments on classification tasks, with a method that writes for regression.
6 - Writing linear interpolations to generate new data points implicitly makes the assumption that the input and output spaces ($\mathcal{X}$ and $\mathcal{Y}$) are convex. I have no clear intuition wether this is a limitation of the authors' proposed method but I strongly feel this should be carefully addressed by a comment in Section 2.