I think it comes down to the scale of the gradients.
In that example, the loss is divided by all words in the mini batch, thus the scale is more and requires a larger LR.
In the previous works in LM normally the loss is only divided by the mini batch size, so LR=1.0 works.