Are we talking about CPU RAM or GPU RAM?
Upon first use of the GPU, PyTorch will initialize CUDA. This part uses (quite a bit) of memory (both GPU and CPU) that won’t be released until PyTorch exists. You could separate this in you analysis by creating a small cuda tensor before loading any model.
For the “user allocated CUDA ram”, using c10::emptyCache() probably is a good idea, also using the caching allocator’s statistics might give you more insight.

Best regards
Thomas