Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, which we call multi-sample dropout, for both accelerating training and improving generalization over the original dropout. The original dropout creates a randomly selected subset (called a dropout sample) from the input in each training iteration while the multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss. This technique can be easily implemented without implementing a new operator by duplicating a part of the network after the dropout layer while sharing the weights among the duplicated fully connected layers. Experimental results showed that multi-sample dropout significantly accelerates training by reducing the number of iterations until convergence for image classification tasks using the ILSVRC 2012 (ImageNet), CIFAR-10, CIFAR-100, and SVHN datasets. Multi-sample dropout does not significantly increase computation cost per iteration for deep convolutional networks because most of the computation time is consumed in the convolution layers before the dropout layer, which are not duplicated. Experiments also showed that networks trained using multi-sample dropout achieved lower error rates and losses for both the training set and validation set.
PAPER SUMMARY: The paper proposes a new and efficient implementation of dropout in which multiple dropout samples are obtained from a single input during training. The authors claim that this enhanced dropout technique (i) accelerates training and (ii) improves generalization by achieving lower error rates than standard dropout in training and validation sets. Experiments in image classification tasks for 4 different datasets (CIFAR-10, CIFAR-100, IMAGENET and SVHN) are presented and the effect of the number of dropout samples and dropout ratio are investigated.
REVIEW SUMMARY: Despite the clear computational advantages of the proposed implementation, this paper should be rejected because  (1) the authors claim multi-sample dropout to be a “new regularization technique”, but the conceptual differences between standard dropout and their proposed method are not clearly stated, (2) one of the main claims regarding “better generalization” is not well supported by their experiments, and (3) the experiments do not consider the stochasticity of the methods they are evaluating.
DETAILED COMMENTS: The multiple-sample dropout proposed by the authors leverages a parallel structure to provide significant acceleration in training, which is supported by experimental results. However, their claims regarding the introduction of a “new regularization technique” and “better generalization” are not convincingly proven in the paper. As mentioned in the paper, the proposed technique of multiple-sample dropout is analogous to the original implementation of dropout, but with larger batch size and duplicated samples. Therefore, the advantage of the proposed method is simply computational efficiency, but not a new “regularization technique”. In addition, while the paper claims the method provides better generalization, the results shown in Figure 6, in which the training losses for their multi-sample dropout and original dropout with duplicated samples are not significantly different, suggest that their validation error might not be significantly different either. The experiments shown in Figure 2, from which they conclude that their method generalizes better, are not conclusive since the validation error is evaluated at a stage in which the original dropout might have not converged, given the slower training. Therefore, the observed differences in validation error might come from the differences in training speed, and not from a better generalization ability of the model. Moreover, in the experiments the authors present single realizations of each training scenario, disregarding the stochastic nature of dropout-based techniques. In particular, the validation error curves observed in Figure 3 (where they analyze the effect of the number of dropout samples) seem very noisy, and the differences observed might simply come from the stochasticity of the training process. The authors should have executed multiple training runs with the same set of hyperparameters, and reported the statistics of their performance (same for results presented in Figure 2 and Table 1).  Overall, the advantages regarding training acceleration and computational efficiency are clear and well supported by the experiments. However, the novelty of the proposed technique and their claim regarding better generalization is not well supported theoretically or experimentally.
For the experiments, the authors should consider:
1)	Report validation errors after convergence of both methods
2)	Run the training scenarios multiple times and report performance statistics instead of single run performance
3)	Report validation curves for experiments in section 3.4 (Figure 6).