Two optimizers for one model 
  
  
    Is there a way to use two optimizers for one model in more beautiful way? 
Now, as I understand we should do something like these: 
net = Model()
part_1_parameters = ...
part_2_parameters = ...
opt1 = optimizer_1(part_1_parameters)
opt2 = optimizer_2(part_1_parameters)

### train epoch
opt1.zero_grad()
opt2.zero_grad()

loss = ...
loss.backward()

opt1.step()
opt2.step()
  



  
    
    
    How to run with multiple GPU when having 2 optimizers? 
  
  
    I think I totally understood the tutorial on using the following line of code to do data parallel. 
model = nn.DataParallel(model) 

I tested it on my device with 2 GPUs and it worked. However, for a complicated model I recently wrote, it doesn’t truly train on both GPUs when I printed the ‘Inside’ ‘Outside’ debugging message. 
I managed to figure out that it was because of the optimizers. To my purpose, I need to have two optimizers each contains some shared part and some unique part. 
model = …
  



  
    
    
    Correct way to declare multiple optimizers for single model in DDP distributed
  
  
    Given the pseudo model below: 
class model(nn.Module):
     def __init__(self):
            self.alpha_params1 = nn.Parameters(<size>, requires_grad=True)
            self.alpha_params2 = nn.Parameters(<size>, requires_grad=True)

           < typical Conv2d layers>....
     
    def forward(self, x):
           <feed forward>
            return output

   def net_parameters(self, filtered_name='alpha_', recurse=True):
          # return torch layer params      
          for name, param in self…
  


check the above discussions