I would suspect that something is up. While the general methodology seems more or less sane (though word splitting is usually replaced by subword splitting these days), it would seem that one would need quite a bit more training to get anywhere. A notebook that does a very similar thing on a char-level basis (much “easier” to train) is in A. Karpathy’s minGPT: minGPT/play_char.ipynb at master · karpathy/minGPT · GitHub That trains a while (40 minutes for me) and then gives output that vaguely looks like Shakespeare…
Best regards
Thomas