I tested your PR with NaN loss, for my case, with adding adapters in mt5 with deepspeed (the same code I shared in the opened issue), I am getting NaN losses, I will be indebted to you to have your advice on things I can try to resolve this issue.
I do not have access to new GPUs and I need to make deepspeed work. Thank you very much for all the incredible jobs you do.