Ah okay. So in short there is no way to do what I want to do; catch and log errors happening in the forward pass during model training, without interrupting training, and while fully utilizing the GPU memory. I.e. there is no way to avoid the extra GPU memory allocation that happens when an error is catched.
Is this something that could be fixed by posting an issue in the pytorch repository?