While you’re filing an Issue, I have cleaned up the code and you can try if it works better for you with this PR [DeepSpeed] improve checkpoint loading code plus tests by stas00 · Pull Request #10760 · huggingface/transformers · GitHub 14
I haven’t tested it with zero3 yet, so it has been tested with zero2 only.
If it solves your problem then great, there is nothing else that you need to do, if not please do file an Issue as suggested in the comment above.