You could use nn.Upsample to manipulate the spatial size of your activation.
However, since your model is pretrained, you might need to finetune it again, as the input to self.conv4 contains now new activations.