6.3.1 Adapters
In recent years, retrofitting adapter modules with a few parameters to pre-trained models has
emerged as an effective approach in speech processing. This involves optimizing the adapter
modules while keeping the pre-trained parameters frozen for downstream tasks. Recent studies (Li
et al., 2023; Liu et al., 2021) [315, 615] have shown that adapters often outperform fine-tuning while
using only a fraction of the total parameters. Different adapter architectures are available, such
as bottleneck adapters (Houlsby et al., 2019)[190], tiny attention adapters (Zhao et al., 2022)[662],
prefix-tuning adapters (Li and Liang, 2021)[314], and LoRA adapters (Hu et al., 2022)[199], among
others Next, we will review the different approaches for parameter-efficient transfer learning. The
different approaches are illustrated in Figure 17 and Figure 18
AReviewofDeepLearningTechniquesforSpeechProcessing