Your are right. The check is for the input, not the target, so I was blind.
The NaNs would also explain the failure of the check, so you would need to debug, where these NaNs are coming from.
Anomaly detection is useful in case the NaNs were created in the backward pass and might be helpful to isolate the issue. Also, check the inputs to the model for invalid values via torch.isfinite just as a sanity check.
If this doesn’t help, check the loss for NaNs and all parameters of the model to further narrow down which operation creates these values.