CUDA OOM is simply when there isn’t enough RAM space on your video card to store all variables.
The variables are your model’s parameters, your data’s physical size, model’s graphs, optimizers’s variables, and various inner parameters PyTorch keeps, plus any unfreed parameters from the previous run.
So to answer your question in order:

calculation of the VGG generates temp parameters to be store in Cuda which can cause OOM.
Unfreed parameters from the previous run added up to cause OOM.
OOM happens randomly due to various reasons, but mostly due to a combination of unfreed parameters, creation of temporary variables for calculation, …etc


a)Reduce your physical batch size, and employ gradient accumulation in software to achieve higher virtual batch size.
b)Be vigilant on how you send data to Cuda. Define a fixed size torch Variable in Cuda and clone your data in CPU to this allocated space to reduce overhead and save space (but might make your model run a teeny bit slower)
c) Take advantage of rtx20xx tensor cores and use fp16 to double your batch size and halve your run time. Take a look at https://github.com/NVIDIA/apex 7.