I found that the imagenet classification example here 3  has a special branch to handle vgg and alexnet which have dense fully connected layers.
After a simple benchmark on VGGNet I found that ONLY parallelizing the model.feature part is obviously faster than parallelizing the whole model.
So what’s the theory behind this behaviour?
Are fully connected layers not friendly to parallelism?