Solved by richard in post #2 
              
              
                new_grid and disp are not leaf nodes of the computation graph so the gradients aren’t accumulated in them. However, you can apply backward hooks to view the gradient at those points.