  Automatic speech recognition (ASR) with federated learning (FL) makes it
possible to leverage data from multiple clients without compromising privacy.
The quality of FL-based ASR could be measured by recognition performance,
communication and computation costs. When data among different clients are not
independently and identically distributed (non-IID), the performance could
degrade significantly. In this work, we tackle the non-IID issue in FL-based
ASR with personalized FL, which learns personalized models for each client.
Concretely, we propose two types of personalized FL approaches for ASR.
Firstly, we adapt the personalization layer based FL for ASR, which keeps some
layers locally to learn personalization models. Secondly, to reduce the
communication and computation costs, we propose decoupled federated learning
(DecoupleFL). On one hand, DecoupleFL moves the computation burden to the
server, thus decreasing the computation on clients. On the other hand,
DecoupleFL communicates secure high-level features instead of model parameters,
thus reducing communication cost when models are large. Experiments demonstrate
two proposed personalized FL-based ASR approaches could reduce WER by 2.3% -
3.4% compared with FedAvg. Among them, DecoupleFL has only 11.4% communication
and 75% computation cost compared with FedAvg, which is also significantly less
than the personalization layer based FL.

Audio and Speech Processing;Computation and Language;Distributed, Parallel, and Cluster Computing;Sound