  Foundation models (FMs) have shown prominent success in a wide range of
tasks. Their applicability to specific domain-task pairings relies on the
availability of, both, high-quality data and significant computational
resources. These challenges are not new to the field and, indeed, Federated
Learning (FL) has been shown to be a promising solution in similar setups. This
paper tackles the specific case of Domain-Adaptive Pre-Training (DAPT), a key
step in the application of FMs. We conduct the first comprehensive empirical
study to evaluate the performance of Federated Domain-Adaptive Pre-Training
(FDAPT). We demonstrate that FDAPT can maintain competitive downstream task
performance to the centralized baseline in both IID and non-IID situations.
Finally, we propose a novel algorithm, Frozen Federated Domain-Adaptive
Pre-Training (FFDAPT). FFDAPT improves the computational efficiency by 12.1% on
average and exhibits similar downstream task performance to vanilla FDAPT, with
general performance fluctuations remaining less than 1%.

Machine Learning;Artificial Intelligence;Distributed, Parallel, and Cluster Computing