Training details. We ﬁrst trained for 10K steps with batch size 32. We used the Adam opti-
mizer (Kingma & Ba, 2014) with parameters β1 = 0.9, β2 = 0.95, (cid:15) = 10−6, and weight decay of
0.1. We considered learning rates in {1 · 10−5, 3 · 10−5, 1 · 10−4, 3 · 10−4} with 0.5% warmup and
linear decay. After initial experimentation on the development set, we set the decay scheduling of the
best neural recursive LM architecture for 100K training steps. We did this for comparability with the
saturated prompt tuning method (see Figure 6) and with textual recursive LMs that trained for this
many steps, though neural-LM recursion does not necessarily converge at the same point as it has
more training parameters.

Results Table 6 presents development set EM scores for all of our ablations and baselines. Overall,
after 10K training steps, LM recursion led to clear gains, with 2.2–3.7 point gains over corresponding

13

Preprint.
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS