Adi_R:

I checked and as far as I can tell I’m not keeping a variable alive for more than it should be.  Explicitly deleting variables does seem to help


This is a sign of holding on to variable for too long. Could you post your code?



 Adi_R:

A follow-up is whether nvidia-smi accurately reports memory usage?


No. PyTorch uses a cached GPU memory allocator. Read more here: http://pytorch.org/docs/master/notes/cuda.html#memory-management 477. Notice that some monitoring methods are not available in a public release yet (they exist on github master).