Hmm yeah. Not really sure how to debug what is happening under the hood. I use Pytorch 1.8.0 and I am running the newest versions of both accelerate and deepspeed. Is this something you can reproduce on your end as well or could it be related to my setup?