You would need to change self.layers too, by inserting your conv4 in the right place. You should add two new keys to the dictionary d (say that your conv4 is something like torch.nn.Conv2d(512, 512, 3)):
d['conv4.0.weight'] = torch.randn((512, 512, 3, 3)) * 0.01
d['conv4.0.bias'] = torch.zeros(512)

Note: be sure to flag these parameters as requires_grad = True, otherwise it won’t learn.