Yes: I’m under impression that there are 2 ways of parallelizing a PyTorch code: DistributedDataParallel and DataParallel. In the former each layer of the network is assigned to a particular processor while in the latter, each processor takes a portion of the training data and all the processors go through all the code (like here). Although DistributedDataParallel is preferred (though I’m not sure why, except for multi-node processing, perhaps?), it looks hairy and I decided to start with DataParallel. Hence, I expected all the processors call first_term() when they get to that part of the code. What am I missing?