I guess, not confirmed yet, that if we save/load a model and this model is directly stored from gpu, the memory is not freed.
I mean, I’ve been working is several clusters and saving model directly from dataparallel module. I’ve had all these troubles loading pretrained weights from this dataparallel state dic.  However I found that saving the model prepared for cpu, loading the model again and then moving it to cuda uses less memory.