Multi-task Learning Multi-task learning (MTL) involves
training on tasks simultaneously. Early work shows that
sharing network parameters across tasks exploits task reg-
ularities, yielding improved performance (Caruana, 1997).
The authors share weights in lower layers of a network,
and use specialized higher layers. Many NLP systems have
exploited MTL. Some examples include: text processing
systems (part of speech, chunking, named entity recogni-
tion, etc.) (Collobert & Weston, 2008), multilingual mod-
els (Huang et al., 2013), semantic parsing (Peng et al., 2017),
machine translation (Johnson et al., 2017), and question an-
swering (Choi et al., 2017). MTL yields a single model
to solve all problems. However, unlike our adapters, MTL
requires simultaneous access to the tasks during training.
Parameter-Efficient Transfer Learning for NLP