My padding mask was incorrect due to some reason. You can try checking for nan values in every step inside your model’s encoder and decoder. If there is any nan value then check for the possible reasons. In my case, my decoder was giving nan values and I changed the implementation and it worked for me.