Upsampling or padding the activations only before concatenation seems to be a bit strange.
Could you add the padding to the conv layers directly and/or pass the upsampled activations to the next layer?
At the moment all layer still see the “smaller” activation, while the padding/upsampling is only performed in the path leading to the concatenation.