AlphaBetaGamma96:

Machine A has 11.0 Cuda installed whereas Machine B is cpu install only! I don’t think that would cause an error?


Since the memory leak seems to be caused on the host memory (if I understand the issue correctly), I don’t think the installed CUDA toolkit version matters here.



 AlphaBetaGamma96:

The leak still exists, so it seems less a difference in PyTorch and perhaps something else that I’m missing.


Yes, that is concerning.



 AlphaBetaGamma96:

Edit: I’ve also been tracking the number of objects within the script via gc.get_objects() and the number of objects increases after the 1st epoch, but it’s constant from the 2nd epoch onward. This behaviour is the same on both machines, although the actual number of objects is slightly lower on the machine that has the leak.


Could you post the machine details (if possible) of the “leaking” machine?
Are you running the workload in a container or on bare metal?
Would it also be possible to get an executable code snippet to debug this issue?
Were you able to check for memory leaks with valgrind?