If you hide from pytorch how the values are computed, it cannot compute the gradient.
I’m not sure what your change_patches function is doing but it looks like you should be able to implement it with regular indexing functions.
Could you give in detail with small example what you want from that function?