A_A:

To clarify further, this feature request actually breaks what I want to do:


Well all the formulas make that assumption so that would most likely break anyways 

No, but I would need grad_z to have N channels coming into backward() for it to compute grad_x.

Then why not fold the padding into your custom function M (making z_g (or just N) an input to the custom Function).
That way the custom function will get the gradient of the right size and can handle all of that whichever wya it wants.