Hi Hyuntae!



 hyuntae:

But when I implement my code, the third network was also be updated…
I think although I did third_optimizer.step(), the grad for the third network should be zero.


If you use weight decay, your optimizer will update the weights even
if their gradients are zero.
(Also, you can easily check whether the gradients are zero.)
Best.
K. Frank