static parameters. The underlying architecture was based on AdapterFusion [102].
Another advancement, AdapterSoup [103], further optimized the adaptation process
by utilizing a weight-averaging approach on domain adapters exclusively during the
evaluation stage. Various techniques for domain adapter selection were investigated,
including exhaustive combination, text clustering, and semantic similarity measures.
Adapters with underlying neural network architectures are commonly referred to
as neural adapters. The seminal design of such adapters is attributed to Houlsby et
al. [98] and consists of a sequential arrangement of down-projection, a GeLU non-
linear activation function [104], and up-projection. These components are integrated
with feed-forward layers to serve as the foundational architecture. Subsequent work
by Bapna et al. [105] streamlined this structure, reducing it to a single hidden-layer
Beyond Efficiency