So I’ve seen the structure of ur network.
U said ur network only accept inputs with size of multiple of 32 so I was just thinking what r the exact multiples of 32 u tested it with (eg: 64, 128, etc)
Have u tried and image of size exactly equal to 32?
Also I’ve not completely studied ur architecture but from what I’m seeing, ur network seems to be downsampling the input with it’s kernals and strides without upsampling back and from what I know about segmentation, the output is usually the same size as it’s image but with annotated pixels 
So what if u feed in an image of 1 x 3 x 64 x 64 what do u get as it’s output.