blade:

Although DistributedDataParallel is preferred (though I’m not sure why)


This is mostly due to performance reasons. As of today, DataParallel (DP) replicates model in its forward pass, while DistributedDataParallel (DDP) replicates models in its ctor. That means DP would replicate model once in every iteration. Besides, DP also suffers from GIL contention as it is single-process-multi-thread. DDP does not hit this problem, as each model replica runs in its own process. More info about DDP can be found here 1 and here 2.

Hence, I expected all the processors call  first_term()  when they get to that part of the code. What am I missing?

What happens in DP’s forward function is: 1) replicate model to all devices 2) scatter inputs to all devices 3) launch multiple threads in parallel, where each threads processes an input split using one model replica on one device 4) gather outputs to the same device.
Given the above, if you change the predictor code to the following. You will see it prints multiple devices.
class predictor(nn.Module):
     ....

    def forward(self, features):
        print(self.first_term(features).device)
        return self.first_term(features) - self.threshold

However, for the following code:
    def SomeFunc(self, model, input_):
        print(model.first_term(input_)[0])

If it is called outside of a forward pass or if the model argument is not a model replica (the self argument in predictor.forward method), then it won’t show different devices.