I am using exactly the code of https://github.com/pytorch/examples/blob/master/imagenet/main.py 25
Moreover, according to Debugging DataParallel, no speedup and uneven memory allocation 32 and https://github.com/pytorch/examples/blob/master/imagenet/main.py#L81 8, the fully connected layers of Alexnet are already implemented on single GPU to overcome synchronization issues.