You may consider to use different optimizers.
It is a way to train generative adversarial networks (training generator network once, discriminator netowork once and so on.)
for example,
model_1 = CNN()
model_2 = Prototype()
optimizer_1 = torch.optim.Adam(
        params=model_1.parameters(),
        lr=lr,
    )
optimizer_2 = torch.optim.Adam(
        params=model_2.parameters(),
        lr=lr,
    )

# during training

if prototype: 
 optimizer_2.step()
else:
 optimizer_1.step()

Is this what you are looking for?
Besides, if you use torch.no_grad, I think it might disconnect gradient chains when loss.backward() is called.
You also can look into the gradients after loss.backward() is called.
loss.backward()
print(model.backbone.conv1.grad) # print(model.some_layer.grad)