While debugging a program with a memory leak I discovered that the leak was bigger when I was using pycharm debugger. I haven’t compared this to other debuggers but there was a definite much larger gpu memory consumption.
I tried a whole bunch of debugger settings, including “on Demand” but none seem to make a difference.
(note: This post has been edited to add this clarification - as I originally blatantly blamed the pycharm debugger, but as @googlebot pointed out in his comments it could be just as well the case with any other debugger. )
the original post follows:

Is anybody using pycharm debugger with pytorch programs? it leaks gpu memory like there is no tomorrow. This sounds similar to the problem with the backtrace on OOM in ipython not freeing gpu memory.
Unknowingly I made the mistake of actually trying to debug memory leakage using pycharm, not realizing that pycharm debugger itself is storing tensors and not freeing them, even with forced gc.collect(). I suppose this is by design since pycharm stores all the variables for the user to access and thus they can’t be freed until the frames are exited.
I discovered that while writing a script to reproduce a memory leak, and only when I added a gpu memory tracing in it using I noticed that that I was getting totally different measurements when running the same script under debugger and not.
I tried a whole bunch of settings, but none seem to help. If you have a way to tell pycharm to not store intermediary vars please share, but somehow I doubt it’s even possible.
So if you’re trying to debug a pytorch program under pycharm and you end up getting OOM, this is why.