You issues are related to the changed activation size, if you manipulate the conv layers.
If you want to keep the output shapes, you would have to set the padding value for your layers, as it’s currently 0 by default.
E.g. for your current setup with kernel_size=5 and stride=1, you would need to add padding=2 to keep the shape.