Roman_Koshkin:

Maybe on the Wikitext2 this model must be trained for days? But then the learning rate should decay much slower. Please help.


It is depending on your hardware, but I would expect training to take a few days, yes.
The perhaps most classic reference with WikiText2 language modelling is Regularizing and Optimizing LSTM Language Models | OpenReview , so I would compare validation perplexities to that to see how you are doing.
Best regards
Thomas