Training neural networks is traditionally done by sequentially providing random mini-batches sampled uniformly from the entire dataset. In our work, we show that sampling mini-batches non-uniformly can both enhance the speed of learning and improve the final accuracy of the trained network. Specifically, we decompose the problem using the principles of curriculum learning: first, we sort the data by some difficulty measure; second, we sample mini-batches with a gradually increasing level of difficulty. We focus on CNNs trained on image recognition. Initially, we define the difficulty of a training image using transfer learning from some competitive "teacher" network trained on the Imagenet database, showing improvement in learning speed and final performance for both small and competitive networks, using the CIFAR-10 and the CIFAR-100 datasets. We then suggest a bootstrap alternative to evaluate the difficulty of points using the same network without relying on a "teacher" network, thus increasing the applicability of our suggested method. We compare this approach to a related version of Self-Paced Learning, showing that our method benefits learning while SPL impairs it.
This problem of interest in this paper is Curriculum Learning (CL), in the context of deep learning in particular. CL refers to learning a non-random order of presenting the training examples to the learner, typically with easier examples presented before difficult ones, to guide learning more effectively. This has been shown to both speed up learning and lead to better generalization, especially for more challenging problems. In this paper, they claim that their contribution is to decompose the problem of CL into learning two functions: the scoring function and the pacing function, with the role of the former being to estimate the difficulty of each training example and the latter to moderate the schedule of presenting increasingly more challenging examples throughout training.
Overall, I found it hard to understand from reading the paper what exactly is new versus what is borrowed from previous work. In particular, after reading Weinshall et al, I realized that they have already proposed a number of things that are experimented with here: 1) they proposed the approach of transfer learning from a previously-trained network as a means of estimating the ‘scoring function’. 2) they also distinguish between learning to estimate the difficulty of examples, and learning the schedule of decreasing difficulty throughout learning, which is actually stated here as the contribution of this paper. In particular, in Section 3 of Weinshall et al, there is a sub-section named “scheduling the appearance of training examples” where they describe what in the terminology of this paper would be called their pacing function. They experiment with two variants: fixed, and adaptive, which are very similar to two of the pacing functions proposed here.
Bootstrapping:
A component of this work that didn’t appear in Weinshall et al, is the bootstrapping approach to estimating the scoring function. In general, this involves using the same network that is being trained on the task to estimate the difficulty of the training examples. The authors explain that there are two ways to do this: estimate how easy each training example is with respect to the ‘current hypothesis’ (the weights of the network at the current step), and with respect to the ‘final hypothesis’, which they estimate if I understand correctly as the network at the end of training. The latter would necessitate first training the network in the standard way, and then using it to estimate how easy or hard each example is, and using those estimates to re-train the network from scratch using that curriculum. They refer to the former as self-paced learning and to the latter as self-taught learning. I find these names confusing in that they don’t really convey what the difference is between the two. Further, while self-paced learning has been studied before (e.g. Kuman et al), I’m not sure about self-taught learning. Is this a term that the authors here coined? If not, it would be useful to add a reference. 
Using easy / hard examples as judged by the current / final hypothesis:
When using the current hypothesis, under some conditions, Weinshall et al showed that choosing harder examples is actually more beneficial than easy examples, similar in spirit to hard negative mining. On the other hand, when using the final hypothesis to estimate examples’ difficulty, using a schedule of increasing difficulty is beneficial. Based on this, I have two comments: 1) It would therefore be useful to implement a version that uses the current hypothesis to estimate how easy each example is (like the self-paced scoring function) but then invert these estimates, in effect choosing the most challenging instead of the easiest ones as is done for anti-curriculum learning. This would be a hybrid between the current self-paced scoring function and anti-curriculum scoring function that would essentially implement the hard negative mining technique in this context. 2) It would be useful to comment on the differences between the self-paced scoring function used here, and that in Kumar et al. In particular, in this case using a curriculum based on this scoring function seems to harm training but in Kumar et al, they showed it actually increased performance in a number of different cases. Why does one work but the other doesn’t?
Experiments:
The experiments are presented in a subset of 5 classes from CIFAR-10 (also used by Weinshall et al.), but also in the full CIFAR-10 and CIFAR-100 datasets. They used both a small CNN (same as in Weinshall et al) as well as a VGG architecture. Overall, their results are comparable to what was previously known: using a curriculum computed by transfer leads to improved learning speed and final performance (though sometimes very slightly) compared to the standard training, and the training with a random curriculum. Further, the benefit is larger when the task is harder (as measured by the final vanilla-trained performance). By computing the distances between the gradients obtained from using a curriculum (via the transfer scoring function) and no curriculum confirms that these two training setups indeed drive the learning in different directions; an analysis similar to Weinshall et al. Also, since, as was previously known and they also observe, the benefit of CL is larger at the beginning of training, they propose a single-step pacing function that performs similarly to other pacing functions while is simpler and more computationally effective. The idea is to decrease only once the proportion of easy examples used in mini-batches, via a step function. Therefore at the start many easy examples are used, and after this threshold is surpassed, few easy examples are used.
 
Overall, I don’t feel the contribution of this paper is large enough to recommend acceptance. The main points that guided this decision are: 
1) The relationship with previous work is not clear. In particular, Weinshall et al seems to have already proposed a few components that are claimed to be the contribution of this paper, as elaborated on above. The authors should mention that the transfer scoring function was borrowed from Weinshall et al, clarify the differences between their pacing functions from those in Weinshall et al., etc. 
2) The usefulness of using easy or hard experiments when consulting the current or final hypothesis is discussed but not explored sufficiently. An additional experiment is proposed above to add another ‘data point’ to this discussion. 
3) self-paced learning is presented as something that doesn’t work and wasn’t expected to work. However, in the past successes were shown with this method, so it would be useful to clarify the difference in setup, and justify this difference.
4) It seems that the experiments resulted to similar conclusions to what was already known. While it’s useful to confirm these findings on additional datasets, I didn’t feel that there was a significant insight gained from them.