A Bayesian Data Augmentation Approach for Learning Deep Models
Data augmentation is an essential part of the training process applied to deep learning models. The motivation is that a robust training process for deep learning models depends on large annotated datasets, which are expensive to be acquired, stored and processed. Therefore a reasonable alternative is to be able to automatically generate new annotated training samples using a process known as data augmentation. The dominant data augmentation approach in the field assumes that new training samples can be obtained via random geometric or appearance transformations applied to annotated training samples, but this is a strong assumption because it is unclear if this is a reliable generative model for producing new training samples. In this paper, we provide a novel Bayesian formulation to data augmentation, where new annotated training points are treated as missing variables and generated based on the distribution learned from the training set. For learning, we introduce a theoretically sound algorithm -generalised Monte Carlo expectation maximisation, and demonstrate one possible implementation via an extension of the Generative Adversarial Network (GAN). Classification results on MNIST, CIFAR-10 and CIFAR-100 show the better performance of our proposed method compared to the current dominant data augmentation approach mentioned abovethe results also show that our approach produces better classification results than similar GAN models.
The authors propose a Bayesian approach to handle Data Augmentation (DA) for learning deep models. Data augmentation, in this context, addresses the problem of data sparsity by generating more training samples to make the training process of a (heavily parameterized) Classifier Network (CN) robust. The standard Generative Model (GM) for obtaining the augmented training set is typically trained once and then new data are generated based on this GM and used to estimate the CN. The authors suggest to also learn the GM itself jointly with the CN as new augmented training data are produced - which is straightforward with a Bayesian approach. Using three different data sets and two classifiers,  they demonstrate that the Bayesian DA is superior to regular DA, which in turn is superior to not having DA at all (which likely suffers from some overfitting).
The authors claim in the introduction that they provide a novel Bayesian formulation of DA. While this may be true, from a Bayesian point of view their idea is quite simple, as this is what any Bayesian statistician would do. Having said that, I think this is a good paper which is clearly written. If it really is the first attempt to think about this problem in a Bayesian way it is by itself worthy of publication, but I am not sufficiently familiar with the deep learning literature to judge this.
The main problem with the pure DA is that one is heavily relying of the GM to be producing correct data, while the proposed approach would allow the GM to be improved. This makes the authors' approach more challenging as both the CN and the GM need to be trained and, in top of this, both are deep models. Hence, making inference using traditional powerful Bayesian tools seems impossible and the authors propose an algorithm which is in spirit with the EM-algorithm to learn the "latents" (here the artificial data) conditional on model parameters and vice versa. Hence my main concern regards the optimization and its robustness to different choices of tuning parameters. For example, Wilson et al. (2017) show that adaptive step-size choices (such as ADAM) are prone to find poor modes in heavily parameterized models. This model is significantly more parameterized than the standard DA, and, although you generate more data, you also have the problem of training the GM. Can you comment on this?  
 
I think that Lemma 1 should be expressed more carefully, stating the assumptions made.
@article{wilson2017marginal,
  title={The Marginal Value of Adaptive Gradient Methods in Machine Learning},
  author={Wilson, Ashia C and Roelofs, Rebecca and Stern, Mitchell and Srebro, Nathan and Recht, Benjamin},
  journal={arXiv preprint arXiv:1705.08292},
  year={2017}
}