Underflow and Overflow Detection
This feature is currently available for PyTorch-only.
For multi-GPU training it requires DDP (torch.distributed.launch).
This feature can be used with any nn.Module-based model.
If you start getting loss=NaN or the model inhibits some other abnormal behavior due to inf or nan in
activations or weights one needs to discover where the first underflow or overflow happens and what led to it. Luckily
you can accomplish that easily by activating a special module that will do the detection automatically.
If you're using [Trainer], you just need to add:
bash
--debug underflow_overflow
to the normal command line arguments, or pass debug="underflow_overflow" when creating the
[TrainingArguments] object.
If you're using your own training loop or another Trainer you can accomplish the same with:
