Dear Stas
I opened up this issue, and I truly appreciate your help https://github.com/huggingface/transformers/issues/10821 12
I had to modify the mt5 model a bit with adding adapter layers which are not yet integrated in huggingface repo, I would be indebted to you if you could have a look and to have your help on this issue.
This is really hard for me to realize the checkpointing issue I encounter with deepspeed and you are my only hope with this issue.
Thank you very much in advance.