the funny part is the mentioned RuntimeError only happens 10% of the time, while the training process could tough it out most of times as if one worker has a slow start (by retraining a second round). I suspect it hits OOM before finish computing first bucket of gradients…