Issue has to do with frozen model weights.
For the same batch size:

10GB gpu memory required for training all resnet101 layers
2.3GB gpu memory required for training only layer4  and fc layers of resnet101 . That’s because it does not need to keep track of gradients for layer1,layer2,layer3 of resnet101.