That could be it… beforehand is
1502MiB /  4043MiB
and if I stop in the middle its
2209MiB /  4043MiB.
Is there a way to clear that up?  I do already have a call to torch.cuda.empty_cache()
.  I also remember reading that with pytorch when GPU memory is freed nvidia-smi will still show it as used even though it can be accessed by future pytorch code.