I also tried not always choosing the top choice, but this only introduces incorrect words and doesn’t break the loop. I

Hm, if you want to generate different texts, you should sample randomly (regarding the probability) – is that what you were doing when you said that you are not always choosing the top choice?
Also, you probably want to reset the hidden state after each generated text.
Note a great one, but I have some simple CharRNN here in case that’s useful: https://github.com/rasbt/stat479-deep-learning-ss19/blob/master/L14_intro-rnn/code/char_rnn.ipynb 116