  Language models (LMs) such as BERT and GPT have revolutionized natural
language processing (NLP). However, the medical field faces challenges in
training LMs due to limited data access and privacy constraints imposed by
regulations like the Health Insurance Portability and Accountability Act
(HIPPA) and the General Data Protection Regulation (GDPR). Federated learning
(FL) offers a decentralized solution that enables collaborative learning while
ensuring data privacy. In this study, we evaluated FL on 2 biomedical NLP tasks
encompassing 8 corpora using 6 LMs. Our results show that: 1) FL models
consistently outperformed models trained on individual clients' data and
sometimes performed comparably with models trained with polled data; 2) with
the fixed number of total data, FL models training with more clients produced
inferior performance but pre-trained transformer-based models exhibited great
resilience. 3) FL models significantly outperformed large language models using
zero-/one-shot learning and offered lightning inference speed.

Computation and Language