UPDATE
I just trained it on my mid-level GPU for about 3 more hours

| end of epoch 186 | time: 71.50s | valid loss  5.71 | valid ppl   302.70


| epoch 187 |   200/ 2928 batches | lr 0.02 | ms/batch 23.74 | loss  3.86 | ppl    47.48
| epoch 187 |   400/ 2928 batches | lr 0.02 | ms/batch 23.63 | loss  3.87 | ppl    47.92
| epoch 187 |   600/ 2928 batches | lr 0.02 | ms/batch 23.60 | loss  3.74 | ppl    41.98
| epoch 187 |   800/ 2928 batches | lr 0.02 | ms/batch 23.67 | loss  3.82 | ppl    45.44
| epoch 187 |  1000/ 2928 batches | lr 0.02 | ms/batch 23.65 | loss  3.84 | ppl    46.41
| epoch 187 |  1200/ 2928 batches | lr 0.02 | ms/batch 23.65 | loss  3.84 | ppl    46.75
| epoch 187 |  1400/ 2928 batches | lr 0.02 | ms/batch 23.63 | loss  3.82 | ppl    45.52
| epoch 187 |  1600/ 2928 batches | lr 0.02 | ms/batch 23.57 | loss  3.87 | ppl    47.81
| epoch 187 |  1800/ 2928 batches | lr 0.02 | ms/batch 23.61 | loss  3.87 | ppl    47.74
| epoch 187 |  2000/ 2928 batches | lr 0.02 | ms/batch 23.65 | loss  3.86 | ppl    47.52
| epoch 187 |  2200/ 2928 batches | lr 0.02 | ms/batch 23.64 | loss  3.72 | ppl    41.37
| epoch 187 |  2400/ 2928 batches | lr 0.02 | ms/batch 23.63 | loss  3.80 | ppl    44.64
| epoch 187 |  2600/ 2928 batches | lr 0.02 | ms/batch 23.62 | loss  3.82 | ppl    45.52
and it seems to improve:
# predict ONE batch

j = 0

src_mask = generate_square_subsequent_mask(bptt).to(device)
data, targets = get_batch(train_data, j) # i кратно bptt
output = model(data, src_mask)


11
, , the ’ the by the ’ s brother , and his doyle that was no evidence to the that be s him the jokes for the brother words jokes — s . with
11
authorities . hornung was angered by doyle ’ s action , and told him there was no need for him to ’ butt in ’ except for his own ’ satisfaction ’ . relations between
11
military authorities . hornung was angered by doyle ’ s action , and told him there was no need for him to ’ butt in ’ except for his own ’ satisfaction ’ . relations

The top paragraph is the prediction on a sample from the training set (targets and inputs are the 2nd and third paragraphs, respectively). Maybe on the Wikitext2 this model must be trained for days? But then the learning rate should decay much slower. Please help.