Instead of resizing (and thus interpolating) you could try to use nn.ConvTranspose2d layers to increase the spatial resolution.
Alternatively, you could also try to keep the spatial size constant throughout the model, but that might not work that well.