marrrcin:

LMs usually have much lower learning rate than 0.001 (i.e BERT has 5e-5), try finding good starting LR by using LR Find algorithm.


For larger scale datasets, yes… but this being a simple toy problem, I expected something higher would work, and it in fact did for a full attention model. I tried again with 1e-4 and it worked a lot better, but still with a small bounce in the training loss.
The Colab is public now.