I don’t think you need multiple optimisers.
You could try to instantiate the optimiser with the parameters that have required_grad = True:
optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=args.lr, weight_decay=args.wdecay)

When you add new layers to your network, you should add the parameters of the new layer via the add_param_group() method.
At this point you will have an optimiser which can optimise the new network (old + new layer).
What’s missing is to ability to freeze the old layers. And I think this can be accomplished just by setting requires_grad=False to the parameters you want to freeze.
Check this as a simple example:
import torch
import torch.optim as optim

w1 = torch.randn(3, 3)
w1.requires_grad = True

o = optim.Adam([w1])
print(o.param_groups)

This prints:
[{'params': [tensor([[-0.7050,  1.0218, -0.7735],
        [-0.0213, -1.2361,  1.2676],
        [-0.8811, -0.5080,  0.4427]], requires_grad=True)], 'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False}]

If then you set w1.requires_grad=False the o.param_groups contains:
[{'params': [tensor([[-0.7050,  1.0218, -0.7735],
        [-0.0213, -1.2361,  1.2676],
        [-0.8811, -0.5080,  0.4427]])], 'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False}]

Also computing the loss twice and backward() twice seems to me a bit strange but I may be missing something. Why don’t you just freeze the layers you don’t want, train on the entire/part of the dataset for one/ more epochs, unfreeze the network and train the whole again.
You are forced to retain the graph with whatever this means (e.g. memory footprint) just because you are computing gradients for the loss twice on the same step