Solved by ptrblck in post #2 
              
              
                The dropout layer doesn’t have trainable parameters, but besides that the code looks alright. 
Since your model is training with the “standard” approach, could you use your per-parameter optimizer with the same learning rate for all parameters, as you’ve used in the working approach? 
Assuming your …