@Oli Hello, thanks for your reply. I just tested, if I set the model to eval() mode, the epoch loss will “jump in a large scale” like I said. And the validation accuracy will stay in a fixed value after about at most 3 epochs. Because I think with torch.no_grad and eval() can accerlerate the training process. But obviously eval() is causing problem and the model stop to train so I remove that. I am not quiet sure why it happens. Because of the size of my training set is quiet small, so I can only take one of the Images as validation set, and I used random cropped patches to train the network, so for the model, all Images are completly new, could the original Images used as validation set ? So that I can use validation loss/accuracy to adjust the model parameter?