That depends what your goal is. If your goal is to get a low res semantics map, then downsample the target. If you want to get high res semantics map, upsample the output (I personally would do convtranspose to upsample).