modeling task was vital to its ﬁne-tuning performance. Doing the same for the prompt tuning baseline
did not seem to similarly help. Using a 2-layered Connector network is better than using a 1-layered
Connector network (perhaps due to the expressive weakness of a single Transformer layer (Cordonnier
et al., 2019)). We did not try deeper Connector networks because of our desire to keep the number of
trained parameters relatively small, but note that this design space is an obvious avenue for future
work.
We continued training the best recursive LM variant (pretrained, 2-layer Connector) until it reached
100K steps, to the point where the prompt tuning baseline had converged and a comparison could be
made when training over the same data as a leading single-pass frozen model method that exhausted
its potential. Table 4 shows this comparison on the Natural Questions test set. Our Neural recursive
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS