I followed the Language Modeling with nn.Transformer and TorchText tutorial exactly, but it doesn’t seem to work at all. After three epochs of training (as is in the tutorial code), the model’s loss is at about 4.0 and the model’s prediction (even on the training set) are gibberish:
# predict ONE batch

j = 0

src_mask = generate_square_subsequent_mask(bptt).to(device)
data, targets = get_batch(train_data, j) # i кратно bptt
output = model(data, src_mask)


i = 6
out = torch.argmax(output, axis=2)[:, i].tolist()
string = ' '.join(vocab.lookup_tokens(out))
print(i)
print(string)

out = targets.reshape(35, 20)[:, i].tolist()
string = ' '.join(vocab.lookup_tokens(out))
print(i)
print(string)

out = data[:, i].tolist()
string = ' '.join(vocab.lookup_tokens(out))
print(i)
print(string)

6
<unk> <unk> <unk> <unk> the <unk> <unk> of <unk> to of the <unk> <unk> the was a a the <unk> <unk> <unk> <unk> <unk> of <unk> to <unk> up to <unk> <unk> the <unk> to
6
<unk> was <unk> for a big loss <unk> losing some of the <unk> that he had gained with the previous play . <unk> state was unable to pick up another first down and sent in
6
<unk> <unk> was <unk> for a big loss <unk> losing some of the <unk> that he had gained with the previous play . <unk> state was unable to pick up another first down and sent

Even if I train the model for 300 epochs, the loss doesn’t go below about 3.6, and the model’s predictions even on the training set are gibberish. So the question is:

Is there a bug in the tutorial code?
Is is true that this model just can’t be expected to produce any sensible predictions, and is only intended to just illustrate how to build a minimalistic encoder-only transformer?
It seems to me that the second answer is more likely, as no example prediction (after training) is provided in that tutorial (as, for example is the case for this other tutorial (LANGUAGE TRANSLATION WITH NN.TRANSFORMER AND TORCHTEXT, in which a transformer (although a different architecture) trained for a few epochs does translate a phrase from German into English. Any help is much appreciated!