Given the pseudo model below:
class model(nn.Module):
     def __init__(self):
            self.alpha_params1 = nn.Parameters(<size>, requires_grad=True)
            self.alpha_params2 = nn.Parameters(<size>, requires_grad=True)

           < typical Conv2d layers>....
     
    def forward(self, x):
           <feed forward>
            return output

   def net_parameters(self, filtered_name='alpha_', recurse=True):
          # return torch layer params      
          for name, param in self.named_parameters(recurse=recurse):   
                if filtered_name not in name:
                      yield param  
   def extra_params(self):
          # return user defined params
          return [self.alpha_params1, self.alpha_params2]         

So above is my pseudo model code.
net = model() # instantiate model above
optimizer_1 = Adam(model.net_parameters, lr=0.001,...)
optimizer_2 = Adam(model.extra_params, lr=0.003,...)
criterion = L1()
##
# typical Apex Distributed Data Parrallel initialization 
##
for epoch in epochs:
  for data1, data2 in dataloader:
       output = net(data1.data)
       loss = criterion(output, data1.gt)
       loss.backward()
       optimizer_1.zero_grad()
       optimizer_1.step()

       output2 = net(data2.data)
       loss2 = criterion(output2, data2.gt)
       loss2.backward()
       optimizer_2.zero_grad()
       optimizer_2.step()
  # save checkpoint
 torch.save(net.module.state_dict(), f"state_dict_{epoch}.pth")
 torch.save(net.module.extra_params(), f"extra_params_{epoch}.pth")
  


Above is my pseudo code for model instantiation and training.
At every 10 epoch intervals, I checkpoint by saving model.state_dict() as well as model’s alpha parameters separately. I then compare the value of my alpha parameters between different epochs. What I found is that both parameters from separate epochs are identical in values as well the model’s weights. It seems no update is happening.  Any help is appreciated.