Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency
<|reference_start|>Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency: Federated Reinforcement Learning (FRL) has garnered increasing attention recently. However, due to the intrinsic spatio-temporal non-stationarity of data distributions, the current approaches typically suffer from high interaction and communication costs. In this paper, we introduce a new FRL algorithm, named $\texttt{MFPO}$, that utilizes momentum, importance sampling, and additional server-side adjustment to control the shift of stochastic policy gradients and enhance the efficiency of data utilization. We prove that by proper selection of momentum parameters and interaction frequency, $\texttt{MFPO}$ can achieve $\tilde{\mathcal{O}}(H N^{-1}\epsilon^{-3/2})$ and $\tilde{\mathcal{O}}(\epsilon^{-1})$ interaction and communication complexities ($N$ represents the number of agents), where the interaction complexity achieves linear speedup with the number of agents, and the communication complexity aligns the best achievable of existing first-order FL algorithms. Extensive experiments corroborate the substantial performance gains of $\texttt{MFPO}$ over existing methods on a suite of complex and high-dimensional benchmarks.<|reference_end|>
@article{yue2024momentum-based,
    title={Momentum-Based Federated Reinforcement Learning with Interaction and
  Communication Efficiency},
    author={Sheng Yue, Xingyuan Hua, Lili Chen, Ju Ren},
    journal={arXiv preprint arXiv:2405.17471},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17471},
    primaryClass={cs.LG cs.AI}
}