Hi @Jun_Wan
Thanks for sharing your code and thanks again for using Opacus!
I looked at your code and I suspect the issue might be because of the learning rate. It seems that the learning rate for the 2 optimizers are different:

on line 155 you define a LR scheduler for optimizer1 but not for optimizer2. So technically the two learnings are going through different training surfaces. You wrote




 Jun_Wan:

Before we call the step() functions, the param.grad are exactly the same between the two models. However, after we called the step() functions, there are approximately a 3% difference between the param.grad of the two models.


and this kind of shows that, I think.


I know the test is for resnet20, but in case you use other resnets, make sure to also include optimizer2 on line 169.


lines 224-228 seem to only be for optimizer1


Last but not least, we have released Opacus 1.0 recently. So, when you have time, please migrate to Opacus 1.0 for more features: Release Opacus v1.0.0 · pytorch/opacus · GitHub
Please let us know if that didn’t solve the issue and we can investigate further.