Solved by ptrblck in post #3 
              
              
                I think kornia is able to differentiate through (some) operations. 
CC @edgarriba, who is one of the main authors of this lib.