Apparently your code has some restrictions on the input shapes, as the height of your current input is 100 and 101 for center and conv4.
Usually it is easier to deal with shapes of powers of two instead of completely random input shapes.
If you really want to use all possible input shapes, you could add some shape checks manually and use cropping or padding to create matching shapes for your concat layers.