showed that our methods match and often exceed prominent ﬁne tuning approaches in massive
multi-tasking and in open-domain question answering. There is clearly much more to learn about
ways in which each of these architectures can be optimized to perform as well as possible in each of
the domains in which we applied it; we have already given pointers to what we consider particularly
low-hanging fruit throughout the paper.
More importantly and more broadly, however, we believe that our results point towards a new stage
in the application of huge LMs to practical problems, in which the design of task-speciﬁc neural
middleware will often take the place of ﬁne tuning. The result will be a landscape in which ﬁne
tuning is typically an unnecessary extravagance and the key engineering challenge is ﬁnding the best
way to stand on the shoulders of giant frozen language models.
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS