Hi all!
I encountered strange GPU behavior and can’t figure why it happens.
I have created five NN’s architectures in a single python kernel. So there are 5 instances of classes net_1, …, net_5 which take same batches as input. One NN takes approximately 600 MB of GPU memory as it is in ‘nvidia-smi’ log. I thought that five would take ~3 GB but they take only ~1 GB… I can save model weights, load them for each model, everything seems to work and learning goes well. But I still can’t explain why additional NN for about 5 mil parameters takes so little space. My guesses were that some strange weight copying happens or… I dunno.
Every model has it separate optimizers obviously, just the flow of learning is the same.
Can anyone explain what’s going on and whether I made a mistake somewhere?
Thanks in advance.