dsr:

The accuracy during training is above 90% but the text that it generates is nonsensical.


RNN/LSTMs are not the current state-of-the-art for natural language generation. For example, in the  Google AI Blog: Reformer: The Efficient Transformer 8 post it says that:

In the language domain, long short-term memory 1 (LSTM) neural networks cover enough context to translate sentence-by-sentence 7. In this case, the context window (i.e., the span of data taken into consideration in the translation) covers from dozens to about a hundred words. The more recent Transformer model 1 not only improved performance in sentence-by-sentence translation, but could be used to generate entire Wikipedia articles through multi-document summarization. This is possible because the context window used by Transformer extends to thousands of words.

Have you tried any of the pre-trained Transformer-based models or trained one of your own?
You may find the following posts and research papers useful:


A Transformer Chatbot Tutorial with TensorFlow 2.0 — The TensorFlow Blog 2


T5: Google AI Blog: Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer (Colab - “Fine-Tuning the Text-To-Text Transfer Transformer (T5) for Closed-Book Question Answering”: Google Colaboratory). arXiv - [1910.10683] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019)


Reformer: Google AI Blog: Reformer: The Efficient Transformer 8


ALBERT: Google AI Blog: ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations


BERT: Google AI Blog: Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing


Fundamentals of Text Generation. This is an introductory tutorial on… | by Murat Karakaya | Deep Learning Tutorials with Keras | May, 2021 | Medium 1 (includes a lot of reference links).