  Large scale language models (LLM) have received significant attention and
found diverse applications across various domains, but their development
encounters challenges in real-world scenarios. These challenges arise due to
the scarcity of public domain data availability and the need to maintain
privacy with respect to private domain data. To address these issues, federated
learning (FL) has emerged as a promising technology that enables collaborative
training of shared models while preserving decentralized data. We propose the
concept of federated LLM, which comprises three key components, i.e., federated
LLM pre-training, federated LLM fine-tuning, and federated LLM prompt
engineering. For each component, we discuss its advantage over traditional LLM
training methods and propose specific engineering strategies for
implementation. Furthermore, we explore the novel challenges introduced by the
integration of FL and LLM. We analyze existing solutions and identify potential
obstacles faced by these solutions within the context of federated LLM.

Machine Learning;Artificial Intelligence;Computation and Language