What I mean is that it should be able to generate text like the RNN does.
I give it a “” token and then I refeed its outputs, when it comes to inference time.
GPT-2 requires a starting phrase, for example.
My dream model would be A kind of Encoder-Decoder model where encoder starts form  and then expands with each generated token in the decoder, while the decoder would be able to use attention mechanisms on the Encoder, or so to say, over the entire generated text.
I hope I explain in such a way to be understandable.