Would it be possible to use DataParallel on just some part of your whole model?
In the ImageNet example 1 only the features are used in parallel for AlexNet and VGG.
In this way you could calculate the scalar values once without averaging for each GPU.
What do you think?