Neural networks can converge faster with help from a smarter batch selection strategy. In this regard, we propose Ada-Boundary, a novel adaptive-batch selection algorithm that constructs an effective mini-batch according to the learning progress of the model. Our key idea is to present confusing samples what the true label is. Thus, the samples near the current decision boundary are considered as the most effective to expedite convergence. Taking advantage of our design, Ada-Boundary maintains its dominance in various degrees of training difficulty. We demonstrate the advantage of Ada-Boundary by extensive experiments using two convolutional neural networks for three benchmark data sets. The experiment results show that Ada-Boundary improves the training time by up to 31.7% compared with the state-of-the-art strategy and by up to 33.5% compared with the baseline strategy.
The paper describes a sampling distribution construction over examples from which to draw mini-batches to train multi-classification models. A distance function on examples is described wherein an example's current (softmax) label probabilities and correctness are taken into account. The bounded distance function supports quantization of example distances and then subsequent sampling from an exponentially decaying probability mass function defined over the binned examples. Results from experiments implementing the proposed method and some baselines on three image classification datasets are provided.
Clearly, any generic improvement to training DNN's has the potential for far-reaching impact. I thought the exposition was fairly clear and appreciated how the introductory sections provided an intuitive understanding of e.g., the differences between the proposed method and the method of Loshchilov and Hutter (2016). The relative conceptual simplicity of the proposed method is a clear positive. The experimental methodology and results are my biggest issues with the paper. The experimental evaluation suggests the proposed method was run 3 times, one for each value of the selection pressure parameter. Then, the best run was selected for comparison. This suggests the proposal is not practical. For results, the benefit of the proposed method is only clearly apparent in one of the three experiments (Fashion MNIST). In the MNIST case, the proposed method does not seem to improve upon the online batch method. For CIFAR-10, where a good case for the proposed method could have been made since the architecture is more complex and potentially more difficult to train, the improvement seems slight. Moreover, it isn't clear whether a relevant baseline was included (see second question below). Also, at least some discussion of computational cost incurred by the method should have been provided. Even better would be to include results wrt/ wall clock training time.
Questions/Comments:
Why not set \gamma = 1? Having a larger value seems to run counter to making training faster. Technically, to use the proposed method, all of the examples need to be processed only once before the distance-based sampling distribution can be utilized. 
Does the random method of the paper denote uniformly at random from the entire dataset per batch or sequential batches from a pre-shuffled dataset per epoch? The distinction is important as Loshchilov and Hutter (2016) report that the latter method performs better than their online batch method on CIFAR-10.
ADA-easy seems to be an irrelevant baseline given the context of the paper.
The phrase "learner's level" is used multiple times, but not defined.
The average is reported in the convergence curves, but shouldn't the variance be reported as well?
Perhaps the selection pressure parameter can be annealed as performed in Loshchilov and Hutter (2016)?