Awesome, glad you found a solution! But it would be ideal for ZeRo to support this natively. I filed an issue as such: [ZeRo] Parameter group support in constructor · Issue #71347 · pytorch/pytorch · GitHub 1