Large code (slighly read it). My guess is you aren’t saving optimizer state_dict.
Adam use running estimators, if you don’t save its state dict, their are initilize while reloading model and just in a single backward leads to wrong point.