Solved by pranayKD in post #6 
              
              
                After spending few hours trying to figure out the problem, found the root cause. And it has nothing to do with model / optimizer loading and saving at all. 
What I needed to change is 
lab_to_val = {j:i for i,j in enumerate(training_classes)}

To 
lab_to_val = {j:i for i,j in enumerate(sorted(traini…