@ptrblck, thanks for the help. I have summarised a list of results which I have obtained. So, basically train() and eval() makes little difference. And changing the momentum value from the default 0.1 to 0.5 also did not seem to make much of a difference. I am sharing the plots of loss and accuracy which I have obtained after training.
Normal VGG16  without batchnorm
 
VGG16  with batchnorm,set to eval()
 
VGG16  with batchnorm, set to train()
 
VGG16  with batchnorm, set to eval() and momentum changed to 0.5
 
VGG16  with batchnorm, set to train() and momentum changed to 0.5
 
So, from the graphs, it’s pretty clear that BatchNorm in general has no improvement irrespective of the momentum value and the mode. Any ideas what could be causing the problem?
I have also shared a link to my Google Colab Notebook which I used for training here 1 if it further helps in debugging anything
Thanks!