I checked all memory is free via nvidia-smi when the system is idle before + after my test, so I dont think other processes are using it.
is it possible that some memory is allocated in the process, but not managed by the PT allocator (e.g., directly goes into the cuda allocator?)