Ratan:

The model architecture is initialized to an object model . This is where the memory fills up.


This would mean that you are already creating CUDATensors inside the __init__ method of your model. Could you check for to('cuda') and cuda() calls inside it?
The next model.cuda() call would then push all remaining parameters and buffers to the GPU and is apparently running out of memory.