@mMagmer thanks a lot for your reply and for catching the typo. I will change the code and let you know the updated loss.
However, is there any effect of the indentation typo with saving the model?
torch.save(model.state_dict(), model_checkpoint_dir)

Because the model (the previous one with indentation typo) performance with the test dataset was terrible.