Supervised deep learning requires a large amount of training samples with annotations (e.g. label class for classification task, pixel-or voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain. During the training of a deep neural network, the annotated samples are fed into the network in a mini-batch way, where they are often regarded of equal importance. However, some of the samples may become less informative during training, as the magnitude of the gradient start to vanish for these samples. In the meantime, other samples of higher utility or hardness may be more demanded for the training process to proceed and require more exploitation. To address the challenges of expensive annotations and loss of sample informativeness, here we propose a novel training framework which adaptively selects informative samples that are fed to the training process. The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model. To evaluate the proposed training framework, we perform experiments on three different datasets, including MNIST and CIFAR-10 for image classification task and a medical image dataset IVUS for biophysical simulation task. On all three datasets, the proposed framework outperforms a random sampling method, which demonstrates the effectiveness of proposed framework.
This paper proposes a method to efficiently select hard samples during the training of a neural network. This is achieved via a variational auto-encoder (VAE) that encodes the samples into a latent space. The VAE is trained in a preparation stage using the images only and fixed at later stage. During training of a DNN framework, samples are selected in the latent space and then decoded via the Decoder in VAE to generate the input for DNN framework. The advantage of such a framework is that now it is able to calculate the gradient w.r.t. the input samples of DNN. This gradient is used to determine the sampling strategy in the next iteration to select harder samples. Two different sampling methods are explored, including nearest neighbor and interpolation (with annotation tool step). The experiments are conducted on small-scale datasets like MNIST CIFAR-10, and IVUS MSE with satisfactory gain over the baselines. Overall, the paper is very well-written and easy to follow. Although the experiment results are not super exciting mainly because of small-scale datasets and not enough gain in the numbers, some of the analysis in Figure 4 are quite insightful to validate the assumption and motivation of this work. So I propose to accept this work for its novelty. I think this work will benefit future research in this direction. 
However, I do have some concerns that I wish the authors could clarify if possible. First, the approach is very similar to online hard negative mining (OHNM) that is purely based on the loss to repeatedly select the samples that generate a larger loss. The major difference is that this work can model the sample distribution and thus select samples based on the gradient w.r.t. the samples in the latent space. This is very novel to me. However, I am wondering if the authors could compare with this sample baseline of OHNM. My concern is that the baselines in this work is too simple and it is not surprise that there is advantage over a simple baseline that is trained without any hard sample mining. 
Second, the experiments are all conducted on small-scale and simple datasets like MNIST and CIFAR10. I am concerned how effective this approach could work for large-scale dataset. In the experiment, even for CIFAR10, a vanilla VAE will not work to reconstruct the input. So the authors have used alpha-GAN to help image reconstruction. If that is the case for CIFAR10 with only 10 classes, how could we extend this work to even larger dataset with more complicated background like ImageNet? I would think the preparation step itself is a very challenging task. This is my major concern that will question the effectiveness of the approach in real applications. 
Third, a related question to the above one. As the input to the DNN is the reconstructed image from the pre-trained decoder, there will be some information loss during the reconstruction process. This is the major challenge, I think, for large-scale applications. Is that possible to use the original image as the input to DNN while still being able to find hard samples using the latent space and the image space correlation? 
Fourth, I really like the visualization of Figure 4 that shows the trajectory of the sampling process that follows the boundaries between classes. The authors also mentioned that some trajectories explore towards outside util there is no real samples, which should be avoided. Could the authors comment on how to avoid such cases? In my understand, as the input is randomly sampled at the beginning, it cannot avoid such cases unless some evaluation is done during training to stop the sampling for these trajectories.