Solved by smth in post #5 
              
              
                just write your loss in terms of autograd operations, and call backward. you dont need to do anything special like writing your own autograd.Function with a custom backward.