Solved by ptrblck in post #2 
              
              
                The loss calculation is independent from the optimization step (parameter update) in general. 
You could create two optimizers with the specified learning rates and apply them sequentially, if that would fit your use case: 
optimizer1.zero_grad()
loss1.backward(retain_graph=True) # calculate gradien…