@peterjc123 - I will try to reproduce the same with simpler models like AlexNet later.
I did try to run the same code on a system with just 4 GB RAM, to see if it gives a out of memory error. Interestingly, it does not give an error and is able to run with just 1 GB free RAM. The same code occupied around 3.5 GBs on a 16 GB system while testing on a single image.