So basically so solve this, I’d have to somehow get the latest “node” in the forward pass before the error occurred, compute a loss with that, and do loss.backward(); optimizer.zero_grad() to make its allocated memory available again for the next batch.
Or always keep half of my GPU memory available for potential “error space” 

On top of that, the GPU memory is cached and is not returned to the OS even though it is available to allocate more Tensors within pytorch.

Ah right.