Existing applications of transformer-based LMs run a given input through the LM only once. While
this is a natural choice, made in most other deep neural network applications, we identify an
opportunity to diverge from this design pattern in the case of LMs. Since both the input and
output spaces of an LM are in natural language, and since the same LM can serve a multitude of
functionalities, it makes sense in principle to re-apply an LM to its own output, an operation which
we dub LM recursion.
In this section we present two distinct methods for putting this idea into practice (see Figure 5)
and give experimental evidence that each of them can produce signiﬁcant gains. In Section 4.1, we
present a textual approach, in which output text is sampled after the ﬁrst pass through the frozen LM
and reinserted into the same frozen LM. In Section 4.2 we present a neural approach, in which a
small trainable network maps the vector representation at the output of the frozen LM to a vector
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS