OK, this would mean the exception is in fact restarting the kernel.
What did you mean by “I can still access all my variables from other cells after error”?
If you still can access and e.g. print old variables, the kernel should be alive and it’s strange that all GPU memory is cleared.
On the other hand, if you cannot print old variables and just restart the training, what would the difference be to a clean restart instead of a restart caused by an exception?