  Federated learning (FL) has been proposed to protect data privacy and
virtually assemble the isolated data silos by cooperatively training models
among organizations without breaching privacy and security. However, FL faces
heterogeneity from various aspects, including data space, statistical, and
system heterogeneity. For example, collaborative organizations without conflict
of interest often come from different areas and have heterogeneous data from
different feature spaces. Participants may also want to train heterogeneous
personalized local models due to non-IID and imbalanced data distribution and
various resource-constrained devices. Therefore, heterogeneous FL is proposed
to address the problem of heterogeneity in FL. In this survey, we
comprehensively investigate the domain of heterogeneous FL in terms of data
space, statistical, system, and model heterogeneity. We first give an overview
of FL, including its definition and categorization. Then, We propose a precise
taxonomy of heterogeneous FL settings for each type of heterogeneity according
to the problem setting and learning objective. We also investigate the transfer
learning methodologies to tackle the heterogeneity in FL. We further present
the applications of heterogeneous FL. Finally, we highlight the challenges and
opportunities and envision promising future research directions toward new
framework design and trustworthy approaches.

Machine Learning