For models with lots of parameters in the classifier (nn.Linear layers at the end) the synchronization would have a large overhead. That’s why it’s often beneficial to use DataParallel on the feature part (conv layers) and compute the linear layers on one GPU.
Have a look at the ImageNet example 43, where this is done for AlexNet and VGG.
Could this be your issue?