Thanks for your help!
Following your suggestion, I found the problem.
During the second training, the model weights are not changing at all.
I need to explicitly add another model.train() step to fix this issue.