Can you provide more details about your training? What type of model? What type of GPUs?
We’re seeing similar issues internally (also involving distributed training). I’m not sure what the problem is yet, but we’re looking into it.