I think I’ve narrowed my problem down to the model itself. I’m using a pretrained resnet and freeze some layers in the process. I think the optimizer is not saving the layers whos training is set to false. I’ll update when I find a more precise solution.