But my model only a few layers need update follow formula 2, other layers update the weights follow formula 1, I think if I change the optim.optimizer, all layers update the weights follow formula 2, that is not what I want.So I think designing a custom layer is possible way to do it. I tried to use the autograd.function, but failed.