Solved by tom in post #2 
              
              
                There isn’t much point in having several optimizers of the same class, as the per-Parameter-computations will be the same. 
If you want different learning rates/other parameters, you can define parameter groups in one optimizer (you can see this e.g. in fast.ai who advocate using different learning …