The right and clean solution would be to create a custom autograd Function for this:
See this page for guidance on writing custom autograd Functions http://pytorch.org/docs/master/notes/extending.html 100