Are you recreating these “zero-padded” convolutions in each iteration or are you zeroing out the gradients?
Did you also verify that you get the same gradients in both approaches?

I tried implementing @ptrblck’s  idea and checking it on a toy example 16. Currently I’m using torch.equal() to compare the gradients. They do not match up in 100% of the cases but almost always. I’m not sure if this is due to some numerical errors or if there is something wrong in my script.