12

Preprint.

1
1
1
1
2
2
2
1

LM Passes LM size Model

7B Prompt-tuned
7B Prompt-tuned
7B LM–Connector
7B Connector–LM
7B LM–Connector–LM
7B LM–Connector–LM
7B LM–Connector–LM
17B Prompt-tuned

Connector Layers Pretrain Init? EM
17.1
17.0
17.3
18.6
18.7
19.6
20.8
22.1

No
Yes
Yes
Yes
No
Yes
Yes
No

–
–
2
2
2
1
2
–

Table 6: Exact match (EM) scores on the development set of Natural Questions after 10K training
steps with batch size 32 of various neural recursive LM ablations and baselines.
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS