I’m guessing that some part of the autocast/grad scaling is attempting to call optimizer.state_dict but ZeRo requires state dict to be consolidated first. This seems like it might be a limitation in terms of autocast and ZeRo working together.
@ kleingeo Could you please file an issue with a full repro to the PyTorch GH repo? Thanks!