Depending on the needed flexibility regarding the input shapes you could either change the conv or transposed conv layers to match the spatial shape e.g. by specifying the output_padding argument or you could use an input shape of powers of 2, which could make this issue a bit easier.