Thanks for your advices. I have detected the bug with the Anomaly detection and torch.isfinite
It’s caused by the cv2.resize(). I use the bilinear interpolation mode to resize the input target. The range of the input to the cv2.resize() is normal, but after the resize operation, the range will beome very very large, which is over the range of float32. Thus, the torch.from_numpy() will return nan or inf.
I replace the bilinear with other mode or use the other resize mthod(transform.resize from skimage). Then everything will run smooth. However, thank you very much