Are you using opt_level='O0' in apex.amp? If so, could you remove apex and run the model in “pure” float32 and check if you are still seeing invalid outputs?
If so, could you check, if the inputs are containing valid values via torch.isfinite(input).all()?