Hmmm,
Soo it’s not all about the gradient. If there is a nan in the forward it will operated with the image features.
Once a NaN appears it expands everywhere. It’s not about the backward nor gradients but the forward.
Which NLP model are you using?
The key is identifiying why are you getting this.
Does it happens always for the same sample(s)?
How are you encoding the text?
You can use a debugger to allow you to have a look once the NaN appears so that you can try to figure out what happened.