Thanks for your patient reply and please forgive my slow response.
I know the idea in your operations, but I don’t know how to implement it. Could you please provide me some specific examples?
By the way, I wonder if it’s possible to manually compute the gradient

image900×91 10.8 KB

By dp_dw[0].backward() and x_1.grad in the first iteration, I can get the gradient  (Is it correct since \Delta W1 is a function of x1 while W1 is not a function of x? ) And it’s easy to get the gradient  with test_loss.backward() and net.parameters().grad in the second iteration. So can I pass the manually computed gradient to a optimizer in order to optimize x1?