@ptrblck, thanks for replying but I think you misunderstood my question. My dataLoader does NOT load all class0 first, then all class1. Each mini-batch contains class0 AND class1, something like this (C0, C0, C0, C1, C1, C1). The weights should be updated based on the average loss. However, this doesn’t result in proper training. Rather, they must be shuffled, e.g. (C0, C1, C0, C1, C1, C0). Is pytorch creating mini-mini-batches from the mini-batches I feed to the model?