Increasing batch size through instance repetition improves generalization
Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling for a fixed budget of optimization steps. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.
This paper proposes increasing the size of the mini-batches by using existing data augmentation methods performed on the same samples of the mini-batch, called Batch Augmentation (BA). The authors claim that this technique provides better generalization. The improvement is justified by reduced variance of the gradient, therefore it might be relevant that the authors compare (computation resources/performances) with Variance Reduced Gradient baselines (see [1]) and analyze it empirically with a better proxy for it (see below). The authors use the L2 norm to empirically analyze the variance of gradients. As the above justification is an important claim, in my opinion using a better proxy for the variance could be useful: for example measuring the variance exactly as done in [1] or using the second-moment estimate as in [2]. As the idea is fairly simple, and the use case seems somewhat limited for a distributed setting, I could be wrong but, in my opinion, the results are not significantly improving: for example in Fig. 4a the baseline outperforms the validation error of BA in the early iterations and marginally down performs at the later iterations. Also, it is not clear to me if this is due to a poor choice of learning rate for the baseline relative to BA (in this case, could be due to using a larger learning rate then its optimal as the validation error is better in the early iterations). - For a more fair comparison, each plot could also compare with a baseline where the batch size is M*B using no data augmentation (e.g. in Fig.1,2,4). - Page 5, 2nd paragraph: the authors mention that increasing M continues the trend of a reduced validation error, whereas from Fig. 3b we see that increasing M starts to saturate the reduction of the validation error. - In my opinion, more detailed ablation studies could be useful. For example, if Adam is used, it would be interesting to see the average effective learning rates for the parameters throughout iterations. Also, a wall-clock comparison on sequential and on distributed settings would be useful. ---Minor--- - Page 4, Sec.2.1: what do the authors mean by “BA reduces variance less” - Page 5, Fig. 4b: the caption is inconsistent [1] On the Ineffectiveness of Variance Reduced Optimization for Deep Learning, A. Defazio and L. Bottou, 2019. [2] Reducing Noise in GAN Training with Variance Reduced Extragradient, T. Chavdarova, G. Gidel, F. Fleuret, S. Lacoste-Julien, 2019.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Increasing batch size through instance repetition improves generalization
Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling for a fixed budget of optimization steps. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes increasing the size of the mini-batches by using existing data augmentation methods performed on the same samples of the mini-batch, called Batch Augmentation (BA). The authors claim that this technique provides better generalization. The improvement is justified by reduced variance of the gradient, therefore it might be relevant that the authors compare (computation resources/performances) with Variance Reduced Gradient baselines (see [1]) and analyze it empirically with a better proxy for it (see below). The authors use the L2 norm to empirically analyze the variance of gradients. As the above justification is an important claim, in my opinion using a better proxy for the variance could be useful: for example measuring the variance exactly as done in [1] or using the second-moment estimate as in [2]. As the idea is fairly simple, and the use case seems somewhat limited for a distributed setting, I could be wrong but, in my opinion, the results are not significantly improving: for example in Fig. 4a the baseline outperforms the validation error of BA in the early iterations and marginally down performs at the later iterations. Also, it is not clear to me if this is due to a poor choice of learning rate for the baseline relative to BA (in this case, could be due to using a larger learning rate then its optimal as the validation error is better in the early iterations). - For a more fair comparison, each plot could also compare with a baseline where the batch size is M*B using no data augmentation (e.g. in Fig.1,2,4). - Page 5, 2nd paragraph: the authors mention that increasing M continues the trend of a reduced validation error, whereas from Fig. 3b we see that increasing M starts to saturate the reduction of the validation error. - In my opinion, more detailed ablation studies could be useful. For example, if Adam is used, it would be interesting to see the average effective learning rates for the parameters throughout iterations. Also, a wall-clock comparison on sequential and on distributed settings would be useful. ---Minor--- - Page 4, Sec.2.1: what do the authors mean by “BA reduces variance less” - Page 5, Fig. 4b: the caption is inconsistent [1] On the Ineffectiveness of Variance Reduced Optimization for Deep Learning, A. Defazio and L. Bottou, 2019. [2] Reducing Noise in GAN Training with Variance Reduced Extragradient, T. Chavdarova, G. Gidel, F. Fleuret, S. Lacoste-Julien, 2019.
assistant
This paper proposes increasing the size of the mini-batches by using existing data augmentation methods performed on the same samples of the mini-batch, called Batch Augmentation (BA). The authors claim that this technique provides better generalization. The improvement is justified by reduced variance of the gradient, therefore it might be relevant that the authors compare (computation resources/performances) with Variance Reduced Gradient baselines (see [1]) and analyze it empirically with a better proxy for it (see below). The authors use the L2 norm to empirically analyze the variance of gradients. As the above justification is an important claim, in my opinion using a better proxy for the variance could be useful: for example measuring the variance exactly as done in [1] or using the second-moment estimate as in [2]. As the idea is fairly simple, and the use case seems somewhat limited for a distributed setting, I could be wrong but, in my opinion, the results are not significantly improving: for example in Fig. 4a the baseline outperforms the validation error of BA in the early iterations and marginally down performs at the later iterations. Also, it is not clear to me if this is due to a poor choice of learning rate for the baseline relative to BA (in this case, could be due to using a larger learning rate then its optimal as the validation error is better in the early iterations). - For a more fair comparison, each plot could also compare with a baseline where the batch size is M*B using no data augmentation (e.g. in Fig.1,2,4). - Page 5, 2nd paragraph: the authors mention that increasing M continues the trend of a reduced validation error, whereas from Fig. 3b we see that increasing M starts to saturate the reduction of the validation error. - In my opinion, more detailed ablation studies could be useful. For example, if Adam is used, it would be interesting to see the average effective learning rates for the parameters throughout iterations. Also, a wall-clock comparison on sequential and on distributed settings would be useful. ---Minor--- - Page 4, Sec.2.1: what do the authors mean by “BA reduces variance less” - Page 5, Fig. 4b: the caption is inconsistent [1] On the Ineffectiveness of Variance Reduced Optimization for Deep Learning, A. Defazio and L. Bottou, 2019. [2] Reducing Noise in GAN Training with Variance Reduced Extragradient, T. Chavdarova, G. Gidel, F. Fleuret, S. Lacoste-Julien, 2019.