Well, for my problem I was doing a 5 fold cross validation using Unet, and what I would do is create a new instance of the model every time and I would create a new instance of the optimizer as well. I assumed it was working because the F1 score would start at 0 each fold when I keep track of the F1 score each epoch.