  In dysarthric speech recognition, data scarcity and the vast diversity
between dysarthric speakers pose significant challenges. While finetuning has
been a popular solution, it can lead to overfitting and low parameter
efficiency. Adapter modules offer a better solution, with their small size and
easy applicability. Additionally, Adapter Fusion can facilitate knowledge
transfer from multiple learned adapters, but may employ more parameters. In
this work, we apply Adapter Fusion for target speaker adaptation and speech
recognition, achieving acceptable accuracy with significantly fewer
speaker-specific trainable parameters than classical finetuning methods. We
further improve the parameter efficiency of the fusion layer by reducing the
size of query and key layers and using Householder transformation to
reparameterize the value linear layer. Our proposed fusion layer achieves
comparable recognition results to the original method with only one third of
the parameters.

Audio and Speech Processing;Sound;Quantitative Methods