Sorry if that explanation wasn’t clear.  I’m not home but this is the psudo code of what was happening.
model = initializeFunction()
optimizer = optimizer(model.params)

do stuff for some epochs

model = loadBestModel()
doMore stuff.

After loadBestModel() I needed to reinitialize the optimizer.  Basically make sure that your optimizer is actually pointing to the parameters that have gradients.