@ptrblck Hello I am sorry for the late reply. After doing debugging for several months, finally I know the main problem. This long time debugging happened because the error is only shows in specific time so I need to run the training again and again to get exactly the error. I wait the error for coming but it just come again several weeks ago. The problem is caused by the Nan value of the prediction. This makes sense that the error not always happened, where its depends on your model performance. Actually the error saying that the value is not between 0 and 1, in fact it is Nan. So I think next time its better to detect the Nan value before calculate the loss. Use pytorch function torch.isnan to make sure the prediction is not Nan. Also I suggest the pytorch should can produce the Nan error instead of only showing error message value not between 0 and 1.