I am not sure to see why you need “a different optimizer” ? What was the first one? You only have a single optimizer over all your parameters here no?
Would the following pseudo code do what you want?
# Outside:
optimizer = optim.smth(net.parameters(), ...)

# Training loop
# Classic fw/bw
out = net.forward(inp)
original_loss = criterion(out, target)
net.zero_grad()
original_loss.backward(create_graph=True)

# compute new loss
new_loss = 0.
for p in net.parameters():
    if not p.grad.eq(0).all():
        new_loss += your_custom_thing(p, p.grad)

# backward new_loss to add it's gradients
new_loss.backward()

# Update your parameters according the gradients for new_loss + original_loss
optimizer.step()