for conditional computation. In this way, CoDA not only
enhances parameter efficiency but also inference efficiency.
Tiny-Attn Adapter (Tiny-Attention Adapter) [18] introduces
a dot-product attention module between the down- and up-
projections, which can also be seen as a multi-head attention
module with its per-head dimensionality to be extremely small.
Moreover, the Tiny-Attn Adapter regards its multiple attention
heads as a mixture of experts and averages their weights to
further reduce inference costs. Akin to the sequential adapter,
the Tiny-Attn Adapter is also injected right after the multi-
head attention layer.
Parameter-EfficientFine-TuningMethods