4. Model explosion. Today’s LM’s zero-shot performance trails that of ﬁne-tuned
models. One can ﬁne-tune the LM to a speciﬁc task, but then lose versatil-
ity. Contemporary eﬀorts to mitigate the problem focus on training a huge
LM jointly on many sets of curated NLP tasks in a massive multi-task setting
(several leading studies reaching 100+ tasks) [6, 7, 15, 16]. These formidable
eﬀorts are eﬀective; the resulting models exhibit versatility and high perfor-
mance when encountering inputs resembling those of the curated tasks. But
the performance of these models on tasks that are not close enough to those
included in the curated tasks can signiﬁcantly deteriorate (for example, per-
plexity degrades signiﬁcantly). It is not practical to ﬁne-tune and serve multi-
ple large models. Nor can one further tune a multi-task-trained LM [6, 7, 15,
16] on a new task that hadn’t been covered in its training; due to catastrophic
MRKL Systems