This should be expected.
Form the docs 11:

The  padding  argument effectively adds  dilation * (kernel_size - 1) - padding  amount of zero padding to both sizes of the input. This is set so that when a  Conv2d  and a  ConvTranspose2d  11 are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when  stride > 1 ,  Conv2d  maps multiple input shapes to the same output shape.  output_padding  is provided to resolve this ambiguity by effectively increasing the calculated output shape on one side. Note that  output_padding  is only used to find output shape, but does not actually add zero-padding to output.

Also, have a look at the conv arithmetic tutorial 43 for more information.