When you are doing backward of grad of grid_sample, you will get an error: derivative for cudnn_grid_sampler_backward is not implemented.
I can compute second order derivative in pure pytorch, but have no idea how to register it, so that it invoked when backward of cudnn_grid_sampler_backward is called.
What is the easiest way to register it?