That’s a neat trick! 
Are you recreating these “zero-padded” convolutions in each iteration or are you zeroing out the gradients?
Did you also verify that you get the same gradients in both approaches?