I actually loaded all my models using DataParallel - so on all the my 4 GPUs.
Each model is ~12M parameters.
I tried to do the inference on the 6 models in the main training loop one after the other and collecting the scores into a tiny model that contains only 1 Linear layer and now my RAM is exploding while my GPUs RAM is releasing.