I haven’t seen this type of error yet, but we will sort it out.
Could you please post it as an Issue 3 and meanwhile I will try to reproduce this. Please tag @stas00 in the issue.
Also please make sure you use the deepspeed master as there are a lot of fixes in there.
When posting tracebacks please always use multiline code formatting which preserves new lines so that it’d be possible to decipher it. Your above formatting is very difficult to understand.
I edited your post to fix the formatting.