I found the issue!.
I was using the pretrained model, so I was loading the weights and optimizers for Adam.
I don’t know why but when I removed the code  and did not load the optimizers in the adam optimizer, the training was okay.
Thank you for your comment.