I just realized that my answer was in a different context … regarding your question, I thought you were referring to Uneven GPU utilization during training backpropagation 36 So my answer may be a bit out of context given the problem discussed in this thread