Hi all,
I’m having trouble recovering from out of memory conditions.  My minibatch routine looks like this:
try:
  nll = computeminibatch(...)
  return nll
except RuntimeError as e:
  optimizer.zero_grad()
  torch.cuda.empty_cache()

This recovery routine works about half the time.  The other half of the time, I crash with an out of memory exception thrown within zero_grad().
Is anyone else seeing this behavior?  Is there a way to clean up and continue without triggering the second OOM condition?
This is pytorch 1.0.1 on python 2.7.15, x86_64, cuda 9.0, cudnn 7.0.4.
Thanks
Jerry