KFrank:

Thanks for this. Could you tell us the values you obtained for a and
b after training?


Even better—if I use the Adam optimizer with its default learning rate of 1e-3, I get full convergence within 30 epochs:
Epoch: 29, Loss: 4.0959913412408344e-12
Final a: 0.5829995274543762, b: 0.11200009286403656

It looks like it really benefits from having per-parameter adaptive learning rates.