I guess the issue is caused by the upsampling layers, which are using transposed convs as well as plain nn.Conv2d layers with overlapping kernels. I would probably try to either remove the nn.Conv2d layers or make sure the kernel do not overlap.