Hi folks,
I’ve seen in the example here that for DataParallel one has to follow the following steps:

Create model
Wrap the model with DataParallel

Send to device

These lead to my first confusion when I tried DistributedDataParallel following the pytorch imagenet example which actually has steps 2 and 3 in reverse order and without that order it throws an error about dense cuda tensors.
My second confusion comes again from following the pytorch imagenet example of DistributedDataParallel usage which utilises torch.utils.data.distributed.DistributedSampler to wrap the train_set but not the valid and test set. Why is that?
My final confusion arises from the fact that running DistributedDataParallel I observe the gpu volatility going up down like it’s the stock market all the way from 0% to 99%, anyone can shed some light on this?