It might be you are seeing an overhead scattering and gathering the whole model.
You could just parallelize the feature layers, if that’s possible.
Have a look at the ImageNet example 20 where this is also performed.
Krizhevsky published this approach in his One weird trick paper 6.