What a transformer can NOT do
&gt;As there is no hidden state, no &quot;memory&quot; or &quot;intention&quot; can be passed to the future<p>This is probably your misconception : &quot;Transformers are RNNs&quot; <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2006.16236.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2006.16236.pdf</a> (see section 3.4)<p>Even though an inner state is not specified explicitly like in a LSTM, for a transformer it&#x27;s possible to view the inner state as implicitly defined as the inner features (deterministically derived from the layers weights) applied to the past inputs.<p>Therefore transformers can learn to do the task by learning to remember a,b,a<i>b to this inner state and write them in order a</i>b,a,b  (Though to train it you&#x27;ll probably have to give it an initial random context to be able to produce different sequences : From this source of entropy it will learn to produce 2 hidden variables which it will use to produce a,b,and a<i>b)<p>I concede that it will not be an easy task to learn in this way, and that it&#x27;s probably easier to train a chain of thought model where you will allow it to use its output to remember the state (by writing it down), instead of having to memorize in its weights. So if you train it to produce a,b,a</i>b,a,b and then another transformer to extract the last 3 outputs a*b,a,b it become very easy.