You could load the state_dict before any manipulation of the first conv layer to make sure all other layers are properly restored.
Once this is done, you could replace the first conv layer with a new one and reassign the parameters manually. This code snippet give you an example:
ref = nn.Conv2d(3, 64, 3, 1, 1)
conv = nn.Conv2d(5, 64, 3, 1, 1)

with torch.no_grad():
    conv.weight[:, :3] = ref.weight
    conv.weight[:, 3:] = torch.zeros_like(conv.weight[:, 3:])
    conv.bias = ref.bias
    
x = torch.randn(2, 3, 24, 24)
out1 = ref(x)
out2 = conv(torch.cat((x, torch.randn(2, 2, 24, 24)), dim=1))

print((out1 == out2).all())
> tensor(True)