U said ur network only accept inputs with size of multiple of 32 so I was just thinking what r the exact multiples of 32 u tested it with (eg: 64, 128, etc)
Have u tried and image of size exactly equal to 32?

Yes I have tried 32,64,224,448 (and some others…)

ur network seems to be downsampling the input with it’s kernals and strides without upsampling back and from what I know about segmentation, the output is usually the same size as it’s image but with annotated pixels 
So what if u feed in an image of 1 x 3 x 64 x 64 what do u get as it’s output.

It might not appear in the lines I have copy/paste above but I can confirm that I have an encoder/decoder architecture (with upsampling in the decoder part). I have actually used a Unet model with the package smp 1. And indeed my output is a mask with exactly the same size as the input (for height and width) …