Is there any resolution to this now? Right now it looks like a single forward pass failing on a single GPU once during training will bring the training crashing down. If anyone has a code example for how to catch these instances and resume training, I’d love to learn of an alternative.