Sorry, I don’t have an answer for that, but…
It looks like a exploding gradients problem
I have two suggestions for you:

take a look on this Image captioning with visual attention  |  TensorFlow Core 2 for some inspiration
There’s this SO answer that might also help you: python - NaN loss when training regression network - Stack Overflow 7


hope it helps