Deep neural networks, which gain great success in a wide spectrum of applications, are often time, compute and storage hungry. Curriculum learning proposed to boost training of network by a syllabus from easy to hard. However, the relationship between data complexity and network training is unclear: why hard example harm the performance at beginning but helps at end. In this paper, we aim to investigate on this problem. Similar to internal covariate shift in network forward pass, the distribution changes in weight of top layers also affects training of preceding layers during the backward pass. We call this phenomenon inverse "internal covariate shift". Training hard examples aggravates the distribution shifting and damages the training. To address this problem, we introduce a curriculum loss that consists of two parts: a) an adaptive weight that mitigates large early punishment; b) an additional representation loss for low weighted samples. The intuition of the loss is very simple. We train top layers on "good" samples to reduce large shifting, and encourage "bad" samples to learn from "good" sample. In detail, the adaptive weight assigns small values to hard examples, reducing the influence of noisy gradients. On the other hand, the less-weighted hard sample receives the proposed representation loss. Low-weighted data gets nearly no training signal and can stuck in embedding space for a long time. The proposed representation loss aims to encourage their training. This is done by letting them learn a better representation from its superior neighbours but not participate in learning of top layers. In this way, the fluctuation of top layers is reduced and hard samples also received signals for training. We found in this paper that curriculum learning needs random sampling between tasks for better training. Our curriculum loss is easy to combine with existing stochastic algorithms like SGD. Experimental result shows an consistent improvement over several benchmark datasets.
This paper suggests a source of slowness when training a two-layer neural networks: improperly trained output layer (classifier) may hamper learning of the hidden layer (feature). The authors call this “inverse” internal covariate shift (as opposed to the usual one where the feature distribution shifts and trips the classifier). They identify “hard” samples, those with large loss, as being the impediment. They then propose a curriculum, where such hard samples are identified at early epochs, their loss attenuated and replaced with a requirement that their features be close to neighboring (in feature space) samples that are similarly classified, but with a more comfortable margin (thus “easy”.) The authors claim that this allows those samples to contribute through their features at first, without slowing the training down, then in later epochs fully contribute. Some experiments are offered as evidence that this indeed helps speedup.
The paper is extremely unclear and was hard to read. The narrative is too casual, a lot of handwaving is made. The notation is very informal and inconsistent. I had to second guess multiple times until deciphering what could have possibly been said. Based on this only, I do not deem this work ready for sharing. Furthermore, there are some general issues with the concepts. Here are some specific remarks.
-	The intuition of the inverse internal covariate shift is perhaps the main merit of the paper, but I’m not sure if this was not mostly appreciated already.
-	The paper offers some experimental poking and probing to find the source of the issue. But that part of the paper (section 3) is disconnected from what follows, mainly because hardness there is not a single point’s notion, but rather that of regions of space with a heterogeneous presence of classes. This is quite intuitive in fact. Later, in section 4, hard simply means high loss. This isn’t quite the same, since the former notion means rather being near the decision boundary, which is not captured by just having high loss. (Also, the loss is not specified.)
-	Some issues with Section 3: the notions of “task” needs a more formal definition, and then subtasks, and union of tasks, priors on tasks, etc. it’s all too vague. The term “non-computable” has very specific meaning, best to avoid. Figure 2 is very badly explained (I believe the green curve is the number of classes represented by one element or more, while the red curve is the number of classes represented by 5 elements or more, but I had to figure it out on my own). The whole paragraph preceding Figure 3 is hard to follow. I sort of can make up what is going, especially with the hindsight of Section 4, since it’s basically a variant of the proposed schedule (easy to hard making sure all clusters, as proxy to classes, are represented) without the feature loss, but it needs a rewriting.
-	It is important to emphasize that the notion of “easy” and “hard” can change along the training, because they are relative to what the weights are at the hidden layer. Features of some samples may be not very separable at some stage, but they may become very separable later. The suggested algorithm does this reevaluation, but this is not made clear early on.
-	In Section 4, the sentence where S_t(x) is mentioned is unclear. I assume “surpass” means achieving a better loss. Also later M_t (a margin) is used, when I think what is meant is S_t (a set). The whole notation (e.g. “topk”, indexing that is not subscripted, non-math mode math) is bad.
-	If L_t is indeed a loss (and not a “performance” like it’s sometimes referred to, as in minus loss), then I assume larger losses means that the weight on the feature loss in equation (3) should be larger. So I think a minus sign is missing in the exponent of equation (2), and also in the algorithm.
-	I’m not sure if the experiments actually show a speedup, in the sense of what the authors started out motivating. A speedup, for me, would look like the training progress curves are basically compressed: everything happens sooner, in terms of epochs. Instead, what we have is basically the same shape curve but with a slight boost in performance (Figure 4.) It’s totally disingenuous to say “this is a great boost in speed” (end of Section 5.2) by saying it took 30 epochs for the non-curriculum version to get to its performance, when within 4 epochs (just like the curriculum version) it was at its final performance basically.
-	So the real conclusion here is that this curriculum may not have sped up the training in the way we expect it at all. However, the gradual introduction of badly classified samples in later epochs, while essentially replacing their features with similarly classified samples for earlier epochs, has somehow regularized the training. The authors do not discuss this at all, and I think draw the wrong conclusion from the results.