Input-dependent prompt-tuning for massively multi-tasking a frozen LM (Section 2). We
show that the gains achieved by ﬁne tuning in the multi-task setup can be achieved with a frozen LM
and a much smaller trained encoder that generates an input-speciﬁc neural prompt. Speciﬁcally, we
slightly exceed the multi-tasking performance of T0++ (Sanh et al., 2021), a ﬁne-tuned 11B-parameter
model, with an input-dependent prompt-tuned J1-Large (Lieber et al., 2021), a frozen 7B-parameter
model. This eliminates the need not only for expensive multi-task ﬁne tuning, but can also reduce
the cost of hosting multiple models. For example, rather than hosting both a massively multi-tasked
model and an LM separately to support different use cases, a single huge LM can be maintained
along with a small encoder that can prompt it per input when needed.
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS