Align as Ideal: Cross-Modal Alignment Binding for Federated Medical Vision-Language Pre-training
<|reference_start|>Align as Ideal: Cross-Modal Alignment Binding for Federated Medical Vision-Language Pre-training: Vision-language pre-training (VLP) has arised as an efficient scheme for multimodal representation learning, but it requires large-scale multimodal data for pre-training, making it an obstacle especially for medical applications. To overcome the data limitation, federated learning (FL) can be a promising strategy to scale up the dataset for medical VLP while protecting data privacy. However, client data are often heterogeneous in real-world scenarios, and we observe that local training on heterogeneous client data would distort the multimodal representation learning and lead to biased cross-modal alignment. To address this challenge, we propose a Federated Align as IDeal (FedAID) framework for federated VLP with robustness to data heterogeneity, to bind local clients with an ideal crossmodal alignment. Specifically, to reduce distortions on global-aggregated features while learning diverse semantics from client datasets during local training, we propose to bind the cross-model aligned representation space learned by local models with an unbiased one via guidance-based regularization. Moreover, we employ a distribution-based min-max optimization to learn the unbiased cross-modal alignment at each communication turn of federated pre-training. The experiments on real-world datasets demonstrate our method successfully promotes efficient federated multimodal learning for medical VLP with data heterogeneity.<|reference_end|>
@article{shuai2024align,
    title={Align as Ideal: Cross-Modal Alignment Binding for Federated Medical
  Vision-Language Pre-training},
    author={Zitao Shuai, Liyue Shen},
    journal={arXiv preprint arXiv:2404.03854},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.03854},
    primaryClass={cs.LG cs.CL cs.CV}
}