Hi, sorry for my delayed answer, but if it can help you, there is a new paper 47 that can help you in your work. They are even available in the Transformers 46 framework. But this work related to QA, not text generation. Quick googling gives me with a pre-trained model from torch hub 
But I kept my project with LSTM as a solution with the transformer failed due to my incorrect usage of the idea of the transformer architecture.