  Parameter-efficient fine-tuning (PEFT) for cross-task generalization consists
in pre-training adapters on a multi-task training set before few-shot
adaptation to test tasks. Polytropon [Ponti et al., 2023] ($\texttt{Poly}$)
jointly learns an inventory of adapters and a routing function that selects a
(variable-size) subset of adapters for each task during both pre-training and
few-shot adaptation. In this paper, we investigate the role that adapter
routing plays in its success and design new variants based on our findings.
First, we build on the intuition that finer-grained routing provides more
expressivity. Hence, we propose $\texttt{MHR}$ (Multi-Head Routing) which
combines subsets of adapter parameters and outperforms $\texttt{Poly}$ under a
comparable parameter budget; by only fine-tuning the routing function and not
the adapters ($\texttt{MHR}$-$z$) we achieve competitive performance with
extreme parameter efficiency. Second, we find that
$\texttt{Poly}$/$\texttt{MHR}$ performance is a result of better multi-task
optimization, rather than modular inductive biases that facilitate adapter
recombination and local adaptation, as previously hypothesized. In fact, we
find that $\texttt{MHR}$ exhibits high gradient alignment between training
tasks. We find that routing is most beneficial during multi-task pre-training
rather than during few-shot adaptation and propose $\texttt{MHR}$-$\mu$, which
discards routing and fine-tunes the average of the pre-trained adapters on each
downstream tasks. This establishes $\texttt{MHR}$-$\mu$ as an effective method
for single-adapter fine-tuning. We also show that $\texttt{MHR}$-$\mu$ can be
used as an effective zero-shot transfer method by training the average of the
pre-trained adapters for a few additional steps on the multi-task training set:
this yields gains up to 3% on absolute accuracy w.r.t. the baselines.

Artificial Intelligence