Thanks for your reply! I want to be a bit cautious, since I’m inexperienced when it comes to Python/PyTorch memory management, but I don’t think it’s the garbage collection mechanism that is responsible for the effects we’ve seen so far. I think I found the underlying issue. Let me explain:

When you wrote that I should play around with the thresholds of the garbage collection, I thought it might be best to completely turn it off. So if the optional garbage collection is enabled, I write at the beginning of my script:

import gc

if __name__ == '__main__': 
    if gc.isenabled(): 
        gc.disable()

However, after some tests, I saw no huge difference from the initially reported results.

What I found is that the statement import torch is quite expensive in terms of memory. For this, I used the following code:

import gc
import torch 
import psutil

if __name__ == '__main__': 
    print('Is Pythons optional garbage collector enabled? Answer: {}'.format(gc.isenabled()))
    if gc.isenabled():
        gc.disable()
        print('Pythons optional garbage collector was disabled.')
    
    resident_memory = psutil.Process().memory_info().rss/1024**2
    print('\nResident memory [MB]: {}'.format(resident_memory))

If I execute the following code on an RTX2080Ti with Cuda10.1, I consistently get as output for the resident memory (this time, I’m not reporting any error bars, but I made sure that the results are not orders of magnitudes apart from each other for different runs):
Resident memory [MB]: 190.37109375

However, the same code executed on an RTX2080Ti with Cuda11.0 yields:
Resident memory [MB]: 981.79296875

For both Cuda versions, I used Python 3.6.9, and for Cuda10.1, PyTorch 1.7.1+cu101, whereas for Cuda11.0, I had PyTorch 1.7.1 available.
My question would be: Why is there such a huge discrepancy of the resident memory  when using different Cuda versions? (The results that I had reported in my initial post were on Cuda version 11.3 if I’m not wrong.)