Solved by Shisho_Sama in post #5 
              
              
                Its not the AverageMeter, since the accuracy drops hugely as well. 
I found the issue . the scheduler needed to be saved and restored upon resuming as well. 
It is explained here