Solved by spanev in post #2 
              
              
                Hi @Hovnatan_Karapetyan, 
Given P1, P2 the two param sets, I think the best way is to create two optimizers with the same hyperparameters: 
optim_even = torch.optim.SGD(P1, lr=0.01)
optim_odd  = torch.optim.SGD(P2, lr=0.01)

for e in range(epochs):
    optim = optim_even if e % 2 == 0 else optim_odd…