Hmm. The issue is that, if a forward pass is interrupted by an error, but the error is not raised, the following forward pass will require additional GPU memory, potentially causing an OOM (if the GPU memory is already almost fully used before the error happens).
In other words, there is no way nicely recover from an error incurred in the forward pass during training, which would be a nice feature.
So just to be clear, you are saying that additional GPU memory is allocated but not actually used?
Were you able to reproduce the problem using a higher batch size?
Thanks for your answers so far by the way, and sorry for my limited understanding