5

(a) Sequential Adapter

(b) Prefix-tuning

(c) LoRA

Fig. 3: The detailed architecture of (a) Sequential Adapter, (b) Prefix-tuning, and (c) LoRA.

knowledge across tasks via hypernetworks while enabling the
model to adapt to each task through task-specific adapters,
significantly reducing the number of trainable parameters.
AdapterSoup [22] is developed to address cross-domain task
adaptation, which first trains multiple adapters based on var-
ious domains and then employs domain clustering [81] to
select the most appropriate top-k adapters for the new domain.
Fine-tuning parameters for the new domain in AdapterSoup
are determined by calculating the weighted average of the
selected k adapters. Apart from cross-domain task adaptation,
AdapterSoup can also be used to strengthen in-domain results
via weight averaging of adapters trained on the same domain
but with different hyperparameters.
Parameter-EfficientFine-TuningMethods