There are plenty of reasons why it could be failing.

Are you using ReLU? If your bias in the convolution are very negative, you could be in the 0 zone of the ReLU
Are you using Batch Norm? It could help with ReLU, or maybe you should check the initialization of your weights?

If you think that the padding is the problem, try padding with F.pad before the convolution, and remove the padding from there, but I doubt it is the cause.