I think your saving and loading is correct.
However, it seems you are comparing the training with the validation accuracy, so I would recommend to add a validation loop before saving and compare it to the restored model.