Thanks for the information! I learned quite a bit from that blog post, so thanks for that.
The issue is however still not resolved. I’ve tried:

putting the try except in a dedicated function

def try_catch_forward(batch, net, optimizer):
   try:
       output = net(batch)
       loss = output["loss"]
       loss.backward()
       optimizer.step()
       optimizer.zero_grad()
   except Exception as e:
       logger.error(e, exc_info=True)
       loss = np.zeros((1))
   gc.collect()
   torch.cuda.empty_cache()
   return loss


manually clearing exception traceback with e.__traceback__= None (in the dedicated function)
doing traceback.clear_frames(e.__traceback__)

not logging the exception at all

gc.collect() and torch.cuda.empty_cache() inside the dedicated try/catch function.

None worked, unfortunately.