Solved by Sia_Rezaei in post #3 
              
              
                @ptrblck Thanks for your reply. I actually figured out the right way to do this. I basically wrote my own Autograd Function class, similar to here: https://pytorch.org/docs/stable/notes/extending.html 
Then inside its backward method, I used a different set of weights to compute grad_input. So the b…