Compressed Federated Reinforcement Learning with a Generative Model
<|reference_start|>Compressed Federated Reinforcement Learning with a Generative Model: Reinforcement learning has recently gained unprecedented popularity, yet it still grapples with sample inefficiency. Addressing this challenge, federated reinforcement learning (FedRL) has emerged, wherein agents collaboratively learn a single policy by aggregating local estimations. However, this aggregation step incurs significant communication costs. In this paper, we propose CompFedRL, a communication-efficient FedRL approach incorporating both \textit{periodic aggregation} and (direct/error-feedback) compression mechanisms. Specifically, we consider compressed federated $Q$-learning with a generative model setup, where a central server learns an optimal $Q$-function by periodically aggregating compressed $Q$-estimates from local agents. For the first time, we characterize the impact of these two mechanisms (which have remained elusive) by providing a finite-time analysis of our algorithm, demonstrating strong convergence behaviors when utilizing either direct or error-feedback compression. Our bounds indicate improved solution accuracy concerning the number of agents and other federated hyperparameters while simultaneously reducing communication costs. To corroborate our theory, we also conduct in-depth numerical experiments to verify our findings, considering Top-$K$ and Sparsified-$K$ sparsification operators.<|reference_end|>
@article{beikmohammadi2024compressed,
    title={Compressed Federated Reinforcement Learning with a Generative Model},
    author={Ali Beikmohammadi, Sarit Khirirat, Sindri Magn'usson},
    journal={arXiv preprint arXiv:2404.10635},
    year={2024},
    doi={10.1007/978-3-031-70359-1_2},
    archivePrefix={arXiv},
    eprint={2404.10635},
    primaryClass={cs.DC cs.LG cs.MA}
}