What could go wrong if I do gradient descent by class? For example, when training on CIFAR10, each minibatch typically contains images from all 10 classes (assuming a moderately large batch size such as 64). What could go wrong if I train on a homogeneous batch of airplane images, followed by a homogeneous batch of automobile images, and so on?
I want to do that in order to calculate some statistics within each class during the training.

Stochastic gradient descent (and variants) works only with the assumption of (in poor words) representative gradients, i.e. the distribution should be almost similar (representative) of your dataset.
Pretty sure that you have no guarantees that SGD converges in that case (might happen however, but you can't know that)
If you come from Statistic, you could see this in this fashion:

SGD uses an estimator of the gradient (which variance depends on which algorithm of the SGD family you pick).
Now, this works only if the estimator is well defined

See "Curriculum Learning"
