summaries: The paper proposes a new federated learning algorithm called SDA-FL, which utilizes GANs to generate synthetic data for federated training on non-IID data. Specifically, each client pretrains a GAN to generate synthetic data and send them to the server. In each round, the server sends the global model and the synthetic data to the clients. Then, the clients update the label of the synthetic data and use both the local data and synthetic data to update the local model. The local models are sent to the server, which further averages the models and uses the averaged model to label the synthetic data. Experiments show that SDA-FL significantly outperforms the other federated learning approaches on non-IID data. The paper also studies the influence of different privacy budgets on the performance when applying differentially private GANs. Federated learning suffers from non-IIDness across clients. A line of previous works address the non-IID problem by data sharing which violates the privacy requirement. In this paper, the author propose SDA-FL. Compare to the most basic FedAvg, the additional components includes  - *Image synthesis with DP-GAN*. Before training classifier, each client trains a differentially private GAN to generate synthetic data, and upload these data to PS for future data-sharing.  - *Synthetic image labeling*. Similar to self-training, PS use local models to assign pseudo-labels to unlabeled data. These labels are updated during training, i.e., interplay between model training and synthetic dataset updating.  - *Mixup*. For each clients, private data and shared synthetic data are mixed to alleviate non-IIDness.  - *ServerUpdate*. Data sharing also makes it possible for the server to conduct gradient descent.   This paper also empirically evaluate their framework under supervised and semi-supervised learning settings. Moreover, this paper study the sensitivity to privacy budget and effect of synthetic data.  This paper focused on a classical federated learning setting where data was non-iid partitioned in different local servers, and came up with a method named SDA-FL, which combined GAN-generated data, differential privacy to both address the non-iid problem and keep local data privacy. The main contributions are: 1. Utilized differential private GAN-generated data to solve the non-iid and local data privacy problems; 2. Designed a label updating mechanism to increase the model performance; 3. Tested the algorithm on CIFAR-10, MNIST, fashion-MNIST to confirm the performance of the algorithm. This paper proposes a new framework for federated learning to resolve the non-IID issue by sharing differentially private synthetic data. Each client pretrains a local GAN to generate synthetic data and upload the data to the parameter server. To effectively use the synthetic data, the server performs pseudo labeling and shares this information with the clients along with the global model parameters, to let the local data, including both the real and synthetic data, approach an IID distribution. The interplay between model training and synthetic data updating improves the convergence of the local models and resolves the non-IID issue. 
abstractText: Federated learning (FL) has recently emerged as a popular privacy-preserving collaborative learning paradigm. However, it suffers from the non-IID (independent and identically distributed) data among clients. In this paper, we propose a novel framework, namely Synthetic Data Aided Federated Learning (SDA-FL), to resolve the non-IID issue by sharing differentially private synthetic data. Specifically, each client pretrains a local generative adversarial network (GAN) to generate synthetic data, which are uploaded to the parameter server (PS) to construct a global shared synthetic dataset. The PS is responsible for generating and updating high-quality labels for the global dataset via pseudo labeling with a confident threshold before each global aggregation. A combination of the local private dataset and labeled synthetic dataset leads to nearly identical data distributions among clients, which improves the consistency among local models and benefits the global aggregation. To ensure privacy, the local GANs are trained with differential privacy by adding artificial noise to the local model gradients before being uploaded to the PS. Extensive experiments evidence that the proposed framework outperforms the baseline methods by a large margin in several benchmark datasets under both the supervised and semi-supervised settings.
references.authors: Manoj Ghuhan Arivazhagan Vinay Aggarwal Aaditya Kumar Singh Sunav Choudhary
references.title: Federated learning with personalization layers
references.venue: arXiv preprint arXiv:1912.00818,
references.authors: Martin Arjovsky Léon Bottou
references.title: Towards principled methods for training generative adversarial networks
references.venue: arXiv preprint arXiv:1701.04862,
references.authors: Christopher Bowles Liang Chen Ricardo Guerrero Paul Bentley Roger Gunn Alexander Hammers David Alexander Dickie Maria Valdés Hernández Joanna Wardlaw Daniel Rueckert. Gan
references.title: augmentation: Augmenting training data using generative adversarial networks
references.venue: arXiv preprint arXiv:1810.10863,
references.authors: Dingfan Chen Ning Yu Yang Zhang Mario Fritz
references.title: Gan-leaks: A taxonomy of membership inference attacks against generative models
references.venue: In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security,
references.authors: Muhammad EH Chowdhury Tawsifur Rahman Amith Khandakar Rashid Mazhar Muhammad Abdul Kadir Zaid Bin Mahbub Khandakar Reajul Islam Muhammad Salman Khan Atif Iqbal Nasser Al Emadi
references.title: Can ai help in screening viral and covid-19 pneumonia
references.venue: IEEE Access,
references.authors: Charles Corbière Nicolas Thome Avner Bar-Hen Matthieu Cord Patrick Pérez
references.title: Addressing failure prediction by learning model confidence
references.venue: In Conference on Neural Information Processing Systems (NIPS),
references.authors: Yuyang Deng Mohammad Mahdi Kamani Mehrdad Mahdavi
references.title: Adaptive personalized federated learning
references.venue: arXiv preprint arXiv:2003.13461,
references.authors: Enmao Diao Jie Ding Vahid Tarokh
references.title: Semifl: Communication efficient semi-supervised federated learning with unlabeled clients
references.venue: arXiv e-prints, pp
references.authors: Moming Duan Duo Liu Xianzhang Chen Yujuan Tan Jinting Ren Lei Qiao Liang Liang
references.title: Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications
references.venue: IEEE 37th international conference on computer design (ICCD),
references.authors: Cynthia Dwork
references.title: Differential privacy: A survey of results
references.venue: In International conference on theory and applications of models of computation,
references.authors: Z. Eaton-Rosen Felix J.S. Bragman Sébastien Ourselin M. Cardoso
references.title: Improving data augmentation for medical image segmentation
references.venue: Conference on Medical Imaging with Deep Learning (MIDL),
references.authors: Avishek Ghosh Jichan Chung Dong Yin Kannan Ramchandran
references.title: An efficient framework for clustered federated learning
references.venue: In Conference on Neural Information Processing Systems (NIPS),
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial networks
references.venue: Communications of the ACM,
references.authors: Ishaan Gulrajani Faruk Ahmed Martı́n Arjovsky Vincent Dumoulin Aaron C Courville
references.title: Improved training of wasserstein gans
references.venue: In NIPS,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: Conference on Conference on Computer Vision and Pattern Recognition
references.authors: Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler Günter Klambauer Sepp Hochreiter
references.title: Gans trained by a two time-scale update rule converge to a nash equilibrium
references.venue: CoRR, abs/1706.08500,
references.authors: Sai Praneeth Karimireddy Satyen Kale Mehryar Mohri Sashank J Reddi Sebastian U Stich Ananda Theertha Suresh
references.title: Scaffold: Stochastic controlled averaging for federated learning
references.venue: In ICML,
references.authors: Kavya Kopparapu Eric Lin
references.title: Fedfmc: Sequential efficient federated learning on non-iid data
references.venue: arXiv preprint arXiv:2006.10937,
references.authors: A Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.venue: Master’s thesis,
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar Virginia Smith
references.title: Federated optimization in heterogeneous networks
references.venue: Proceedings of Machine Learning and Systems (MLSys),
references.authors: Xiang Li Kaixuan Huang Wenhao Yang Shusen Wang Zhihua Zhang
references.title: On the convergence of fedavg on non-iid data
references.venue: In International Conference on Learning Representations,
references.authors: Paul Pu Liang Terrance Liu Liu Ziyin Nicholas B Allen Randy P Auerbach David Brent Ruslan Salakhutdinov Louis-Philippe Morency
references.title: Think locally, act globally: Federated learning with local and global representations
references.venue: arXiv preprint arXiv:2001.01523,
references.authors: Tao Lin Lingjing Kong Sebastian U Stich Martin Jaggi
references.title: Ensemble distillation for robust model fusion in federated learning
references.venue: In Conference on Neural Information Processing Systems (NIPS),
references.authors: H. Brendan McMahan Eider Moore Daniel Ramage Seth Hampson Blaise Agüera y Arcas
references.title: Communication-efficient learning of deep networks from decentralized data
references.venue: International Conference on Artificial Intelligence and Statistics (AISTATS),
references.authors: Mehdi Mirza Simon Osindero
references.title: Conditional generative adversarial nets
references.venue: arXiv preprint arXiv:1411.1784,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.authors: Dinh C Nguyen Ming Ding Pubudu N Pathirana Aruna Seneviratne Albert Y Zomaya
references.title: Federated learning for covid-19 detection with generative adversarial networks in edge cloud computing
references.venue: IEEE Internet of Things Journal,
references.authors: Augustus Odena Christopher Olah Jonathon Shlens
references.title: Conditional image synthesis with auxiliary classifier gans
references.venue: In International conference on machine learning,
references.authors: Seungeun Oh Jihong Park Eunjeong Jeong Hyesung Kim Mehdi Bennis Seong-Lyun Kim
references.title: Mix2fld: Downlink federated learning after uplink federated distillation with two-way mixup
references.venue: IEEE Communication Letters,
references.authors: Sinno Jialin Pan Qiang Yang
references.title: A survey on transfer learning
references.venue: IEEE Transactions on knowledge and data engineering,
references.authors: Alec Radford Luke Metz Soumith Chintala
references.title: Unsupervised representation learning with deep convolutional generative adversarial networks
references.venue: arXiv preprint arXiv:1511.06434,
references.authors: Daniel Rothchild Ashwinee Panda Enayat Ullah Nikita Ivkin Ion Stoica Vladimir Braverman Joseph Gonzalez Raman Arora
references.title: Fetchsgd: Communication-efficient federated learning with sketching
references.venue: In International Conference on Machine Learning,
references.authors: MyungJae Shin Chihoon Hwang Joongheon Kim Jihong Park Mehdi Bennis Seong-Lyun Kim
references.title: Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML 2020 (FL-ICML’20), 2020
references.authors: Kihyuk Sohn David Berthelot Nicholas Carlini Zizhao Zhang Han Zhang Colin A Raffel Ekin Dogus Cubuk Alexey Kurakin Chun-Liang Li
references.title: Fixmatch: Simplifying semi-supervised learning with consistency and confidence
references.venue: Advances in Neural Information Processing Systems,
references.authors: Reihaneh Torkzadehmahani Peter Kairouz Benedict Paten
references.title: Dp-cgan: Differentially private synthetic data and label generation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Hao Wang Zakhary Kaplan Di Niu Baochun Li
references.title: Optimizing federated learning on non-iid data with reinforcement learning
references.venue: In IEEE INFOCOM 2020-IEEE Conference on Computer Communications,
references.authors: Jianyu Wang Qinghua Liu Hao Liang Gauri Joshi H Vincent Poor
references.title: Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 2020b
references.authors: Han Xiao Kashif Rasul Roland Vollgraf
references.title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
references.venue: arXiv preprint arXiv:1708.07747,
references.authors: Liyang Xie Kaixiang Lin Shu Wang Fei Wang Jiayu Zhou
references.title: Differentially private generative adversarial network
references.venue: arXiv preprint arXiv:1802.06739,
references.authors: Tehrim Yoon Sumin Shin Sung Ju Hwang Eunho Yang
references.title: Fedmix: Approximation of mixup under mean augmented federated learning
references.venue: arXiv preprint arXiv:2107.00233,
references.authors: Naoya Yoshida Takayuki Nishio Masahiro Morikura Koji Yamamoto Ryo Yonetani
references.title: Hybridfl for wireless networks: Cooperative learning mechanism using non-iid data
references.venue: IEEE International Conference on Communications (ICC),
references.authors: Felix Yu Ankit Singh Rawat Aditya Menon Sanjiv Kumar
references.title: Federated learning with only positive labels
references.venue: In International Conference on Machine Learning,
references.authors: Hongyi Zhang Moustapha Cisse Yann N. Dauphin David Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.venue: International Conference on Learning Representations (ICLR),
references.authors: Longling Zhang Bochen Shen Ahmed Barnawi Shan Xi Neeraj Kumar Yi Wu
references.title: Feddpgan: Federated differentially private generative adversarial networks framework for the detection of covid-19 pneumonia
references.venue: Information Systems Frontiers,
references.authors: Yue Zhao Meng Li Liangzhen Lai Naveen Suda Damon Civin Vikas Chandra
references.title: Federated learning with non-iid data
references.venue: arXiv preprint arXiv:1806.00582,
references.authors: Hangyu Zhu Jinjin Xu Shiqing Liu Yaochu Jin
references.title: Federated learning on non-iid data: A survey
references.venue: arXiv preprint arXiv:2106.06843,
sections.heading: 1 INTRODUCTION
sections.text: The recent development of deep learning technologies has led to major breakthroughs in various domains. This results in a tremendous amount of valuable data that can facilitate the training of deep learning models for intelligent applications. A traditional approach to exploit these distributed data samples is to upload them to a centralized server for model training. However, directly offloading data raises severe privacy concerns as data collected by mobile clients may contain sensitive information.
  By decoupling model training from the need of transferring private data to the cloud, federated learning (FL) offers a promising approach to collaboratively learn a global model without directly sharing the local data. Particularly, McMahan et al. (2017) introduced the Federated Averaging (FedAvg) algorithm that synchronously aggregates the updates of local models, while satisfying the basic requirements for privacy protection. Later, FedProx was proposed by Li et al. (2020) that adds a proximal term in the loss function to improve the statistical stability of the training process.
  A key characteristic of federated learning is the non-IID (independent and identically distributed) data distribution among clients. Consequently, different clients learn from different data distributions, which leads to high heterogeneity among local models and degrades the effectiveness of model aggregation (McMahan et al., 2017; Wang et al., 2020b). For extreme cases with highly skewed data distribution, traditional federated learning methods even lack a theoretical guarantee of convergence (Yu et al., 2020; Rothchild et al., 2020). Recent studies have attempted to address the non-IID issue by constructing a global shared dataset based on the local data to mitigate the data imbalance effect (Krizhevsky et al., 2009; Zhao et al., 2018; Yoshida et al., 2020; Oh et al., 2020). Particularly, several methods (Krizhevsky et al., 2009; Yoshida et al., 2020) proposed data sharing strategies to distribute a percentage of raw data among clients, but naive data sharing seriously violate the privacy requirement in FL. Besides, data augmentation approaches (Zhang et al., 2021; Oh et al., 2020) generate synthetic samples without directly sharing private data. However, these methods are still susceptible to data leakage without a privacy-preserving mechanism. Thus, it is important to investigate how
  to generate privacy-preserving synthetic samples, as well as how to effectively utilize the private dataset and synthetic dataset in the FL framework.
  To alleviate the heterogeneity of the data distribution in FL without compromising privacy, we propose a novel framework, named Synthetic Data Aided Federated Learning (SDA-FL), which resolves the non-IID issue by sharing differentially private synthetic data. Under this framework, each client pretrains a local generative adversarial network (GAN) (Goodfellow et al., 2020) to generate synthetic data, thus avoiding sharing the raw data directly. A parameter server (PS) performs pseudo-labeling for the uploaded synthetic data and constructs a global shared dataset. In each global aggregation, the PS updates the pseudo labels of the synthetic dataset based on the received models and updates the global model with the confident synthetic data. This iterative labeling process constitutes a synthetic dataset with high-confidence labels, which is fed back to the clients to construct their training samples, including the local private data and synthetic data, following an almost identical distribution. Meanwhile, to provide a privacy guarantee for the generative models, we adopt the widely used differential privacy (Dwork, 2008) to ensure that the actual training samples can not be revealed by adversaries. In particular, we use the Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017) as the backbone and introduce the privacy budget by adding artificial noise to the gradients in the training process. The SDA-FL framework is mostly compatible with many existing FL methods and can be applied in both supervised and semi-supervised settings.
  We evaluate the proposed framework on several standard datasets under both supervised learning and semi-supervised learning, which shows its effectiveness in resolving the non-IID distribution problem even in the extreme case (with only one class of real data at each client). We also investigate the tradeoff between the privacy budget and the FL performance, which demonstrates that this framework is fully functional under the strict differential privacy protection requirement. Our major contributions are summarized as follows:
  • We develop a global shared synthetic dataset by the differentially private GAN models. This data augmentation method effectively resolves the non-IID problem in FL and maintains the local data privacy.
  • To utilize the global shared synthetic dataset efficiently, we propose a novel framework, SDA-FL, in which the PS provides and keeps updating high-confidence pseudo labels for the synthetic dataset iteratively, thus improving the performance of the global model.
  • We evaluate our framework on several benchmark datasets under the high heterogeneous data distribution, where our framework consistently outperforms other baselines under both supervised and semi-supervised learning. Besides, our framework experiences a slight performance degradation with a stringent privacy budget.
sections.heading: 2 RELATED WORK
sections.text: Non-IID Challenges in Federated Learning The pioneering work in FL is FedAvg (McMahan et al., 2017), which simply averages model parameters trained by different clients in an elementwise manner, weighted proportionately by the clients’ data size. However, this algorithm often leads to slow convergence and performance degradation when data is heterogeneous across local clients (Li et al., 2019). To alleviate this issue, Wang et al. (2020a) leveraged the reinforcement learning to intelligently choose the client devices to participate in each round of federated learning to counterbalance the bias introduced by non-IID data. Besides, some studies proposed personalized layers to adjust the local model according to the private dataset (Arivazhagan et al., 2019; Liang et al., 2020). For example, FedPer was introduced in Arivazhagan et al. (2019) that captures personalization aspects in federated learning by only using the local data to train the personalized layers. Knowledge distillation (Deng et al., 2020; Lin et al., 2020) is also a promising idea of personalized federated learning, which transfers information from other clients to a certain client. Although the aforementioned methods outperform FedAvg on strongly non-IID data, the extra computational costs are not affordable by the resource-constrained IoT devices (Zhu et al., 2021). Thus, client clustering has been proposed to construct a multi-center framework by grouping the clients into different clusters based on the model similarity (Ghosh et al., 2020; Kopparapu & Lin, 2020). As clients avoid the negative knowledge transfer from the dissimilar models (Pan & Yang, 2009), clustering methods can alleviate the negative effect of the non-IID data distribution. However, such methods are not scalable as the communication cost increases largely with the number of clients.
  Data Augmentation and Privacy Preserving Recently, methods based on some form of data sharing have received increasing attentions and achieved prominent performance in FL (Zhu et al., 2021; Zhao et al., 2018). Particularly, data augmentation approaches showed the potential capability to address the non-IID issue by modifying the local distributions without sharing private data (Duan et al., 2019; Zhang et al., 2018; Eaton-Rosen et al., 2020). A vanilla data augmentation method was proposed by Duan et al. (2019), in which each client performs local data augmentation to reach a balanced global distribution. Besides, several methods (Oh et al., 2020; Shin et al., 2020; Yoon et al., 2021) leveraged the Mixup technique (Zhang et al., 2018) to conduct data augmentation, where the clients share the blended local data and collaboratively construct a new global dataset to tackle the non-IID issue. However, frequent data exchange may be susceptible to privacy issues. Recently, the GAN-based data augmentation framework (Bowles et al., 2018; Yoshida et al., 2020) was shown to be promising for reducing the degree of local data imbalance in FL. The general idea is to train a good generative model in the presence of non-IID data. Under this framework, each client sends its local seed samples to the server for the GAN training, and then a well-trained generator will be sent to all the clients for model updating. However, directly sending local data samples to the server violates the data privacy requirement. FedDPGAN (Zhang et al., 2021) suggested an alternative that all the clients train a global GAN together based on the FL framework to supplement the scarce local Covid-19 data. Unfortunately, the white-box generators sent to the server are vulnerable to adversarial attacks (Chen et al., 2020).
sections.heading: 3 PRELIMINARY
sections.text: Federated Learning FedAvg (McMahan et al., 2017) is one of the representative training algorithms in federated learning, and it trains a shared neural network without disclosing the raw data of clients. For every step t = 0, . . . , T − 1, every client k ∈ St downloads the global model wt and updates the local model with local dataset Dk via stochastic gradient descent (SGD), i.e., wt+1
  k ← wtk − ηt+1∇`. The updated local models are then sent back to the PS for a weighted aggregation wt+1 ← 1K ∑ k∈St w k t+1. These procedures repeat until convergence or all the T training steps are exhausted.
  Differentially Private Generative Adversarial network We utilize a differentially private generative adversarial network (DPGAN) for data augmentation while protecting local data privacy. To avoid the gradients vanishing and mode collapse problems (Arjovsky & Bottou, 2017; Arjovsky et al., 2017) that are commonly encountered by many GAN models (Mirza & Osindero, 2014; Radford et al., 2015; Odena et al., 2017), we adopt the WGAN-GP (Gulrajani et al., 2017), which is upgraded based on the Wasserstein GAN (WGAN) (Arjovsky & Bottou, 2017; Arjovsky et al., 2017), as the backbone of the generative model for a stable training procedure. Besides, we add noise to the gradient in the generator training process, which provides privacy control in terms of differential privacy (Torkzadehmahani et al., 2019). The definition of differential privacy is as follows: Definition 1. (Differential privacy1): A random mechanism Ap satisfies ( ,δ)-differential privacy if for any output’s subset (S) and for any two adjacent datasets D, D′, the following probability inequality holds:
  P (Ap(D) ∈ S) ≤ e · P (Ap (D′) ∈ S) + δ. (1)
  Particularly, δ is an optimal probability that guarantees the validity of the inequality, and is the privacy budget indicating the privacy level, i.e., a smaller value of implies stronger privacy protection. To satisfy the ( , δ)-differential privacy, we follow the work (Xie et al., 2018) and add the Gaussian noise to the gradients updated in each discriminator training iteration. The relationship between the noise variance and differential privacy is shown as follows:
  σn = 2q √ nd log ( 1
  δ
  ) / , (2)
  where q and nd denote the sample probability and the number of discriminator training iterations in each training round, respectively.
  1Although our work leverages the differential privacy to address the extra privacy problem introduced by the GAN-based data augmentation, the potential privacy risk of FL is still an open problem, which is out of the scope of our study.
  Pseudo Labeling Although GAN-based data augmentation can generate synthetic data, the labels of the synthetic data are unknown, which hinders the training of the model through backpropagation. In semi-supervised learning, pseudo labeling is a widely used method in classification tasks to generate confident pseudo labels (Sohn et al., 2020). A hard pseudo label will be applied to the unlabeled data if the maximum class probability (MCP) of the inference result (Corbière et al., 2019) exceeds a predefined threshold τ . In our experiments, the threshold is set to be 0.95.
sections.heading: 4 SYNTHETIC DATA AIDED FEDERATED LEARNING (SDA-FL)
sections.text: We now introduce the SDA-FL framework that adopts GAN-based data augmentation to alleviate the negative effect of the non-IID data. The system diagram is shown in Fig. 1, and key algorithmic innovations built upon the classic FL framework are elaborated below.
  Global Synthetic Dataset Construction At the beginning of the federated training, each client pretrains a local GAN model to generate synthetic samples based on its private data. Then, the synthetic samples are sent to the PS to construct a global shared synthetic dataset. To effectively leverage the synthetic dataset for federated learning, we perform pseudo labeling for these samples, which is critical for the effectiveness of the proposed framework.
  In federated learning, the local model is the most suitable network to predict the pseudo labels for the synthetic data generated by the corresponding local GAN, as they are both trained with the same data. In our framework, to improve the confidence level of the pseudo labels, we utilize the local models to perform the pseudo labeling for the unlabeled synthetic data. Specifically, after receiving the local model wit+1 in each aggregation step, the PS utilizes the model predictions to assign a pseudo label for each unlabeled synthetic instance if the maximum class probability is higher than a predefined threshold τ . In this way, we are gradually generating synthetic data samples with high-quality labels.
  Synthetic Data Aided Model Training Augmented by the samples from the shared synthetic dataset that are with the confident labels, the data available for local training at different clients reach approximately homogeneous distribution. To make good use of the synthetic data, we leverage the Mixup method (Zhang et al., 2018), which utilizes linear interpolation between the real sample (xi, yi) and the synthetic sample (x̂i, ŷi) to augment the real data:
  x̄ = λ1x̂i + (1− λ1)xi, ȳ = λ1ŷi + (1− λ1)yi,
  (3)
  where λ1 follows the Beta distribution ( Beta(α, α) ) with α ∈ [0, 1]. Combined with the crossentropy loss, the Mixup loss for local update becomes:
  `1 = λ1` ( f(X̄;wi), Ŷi ) + (1− λ1)` ( f(X̄;wi),Yi ) . (4)
  Algorithm 1: Synthetic Data Augmented Federated Learning (SDA-FL) Input: Local dataset Dk = {Xk,Yk},
  Pretrained generators G = {Gk} k ∈ {1, . . . , N}
  Initialize w0,Synthetic dataset D̂s = {X̂s, Ŷs} for server, D̂k = {X̂k, Ŷk} for clients All clients upload the synthetic data generated by Gk to PS for t = 0, . . . , T − 1 do St ← K clients are selected at random Send wt and updated pseudo labels to
  clients k ∈ St Clients Side: for k ∈ St do
  Updata local synthesis dataset(X̂k, Ŷk) wkt+1 ← LocalUpdate(k,wt;Xk,Yk, X̂k, Ŷk)
  end Parameter Server Side: Pseudo Label Update wt+1 ← 1K ∑ k∈St w k t+1
  wt+1 ← ServerUpdate(wt+1; X̂s, Ŷs) end
  Algorithm 2: Model Update
  LocalUpdate(k,wt;Xk,Yk, X̂k, Ŷk) : w ← wt for e = 0, . . . , E − 1 do
  Split Dk into batches of size B for batch(X,Y ) do
  Sample batch(X̂, Ŷ ) from (X̂k, Ŷk)
  Use (X̂, Ŷ ) and (X,Y ) to update w following (6)
  end end return w ServerUpdate(wt+1; X̂s, Ŷs): w ← wt+1 for e = 0, . . . , E − 1 do
  Sample batch(X̂, Ŷ ) from (X̂s, Ŷs) Evenly divide (X̂, Ŷ ) into (X̂1, Ŷ1)
  and (X̂2, Ŷ2) Use (X̂1, Ŷ1) and (X̂2, Ŷ2) to update w following (6)
  end return w
  In addition, to keep more information of the local clients, another cross-entropy loss term is introduced for the real data as follows:
  `2 = ` ( f(Xi;wi),Yi ) . (5)
  Then SGD is applied to update the local model:
  wi ← wi − ηt+1∇(`1 + λ2`2), (6) where λ2 is a hyperparameter to control the retention of the local information and will be set in different values in experiments. More details about the local update are shown in Algorithm 2.
  In contrast to the traditional federated learning where the PS does not have access to any data to update the global model, the PS in our framework can keep the entire global synthetic dataset D̂s and uses it to update the global model, thereby helping to enhance the performance.
  Interplay between Model Training and Synthetic Dataset Updating Because of the non-IID problem, there is a distinct difference in the weights of the local models among clients in federated learning. Furthermore, simply sharing the synthetic data with the PS and other clients will not result in the good training performance of the global model due to the lack of confident pseudo labels. Our framework, SDA-FL, deeply integrates federated learning and synthetic data, which effectively alleviates the non-IID issue and helps to train a good global model. As shown in Fig. 1, after creating the global synthetic dataset, it will be shared with all the clients so that the local data, including both the real and the synthetic data, approach an IID distribution. Without a doubt, the supplement of synthetic data results in the improvement of the local models. After being uploaded to the PS, the updated local models are used to perform the pseudo labeling and update the global synthetic dataset by the PS. The stronger local models can boost the confidence of the pseudo label. With the improvement of labeling for the global synthetic dataset, the PS can utilize it to update the global model and all the clients can improve their local models subsequently at the next-round training. Thus there is an interplay between the model update and the synthetic dataset at every training round: global synthetic dataset → global model update → local model update → labeling for global synthetic dataset, which results in a converged and well-perform model.
  SDA-FL vs. Traditional FL From the above descriptions, the proposed SDA-FL framework introduces two additional operations at the server side, namely, pseudo label update of the shared synthetic dataset and server-side model update, as shown in Fig. 1. These two main innovations contribute to the performance improvement with non-IID data. In traditional FL algorithms (Li et al., 2020; Karimireddy et al., 2020), clients update their models based on local private data, which may lead to performance degradation when data are heterogeneous over clients. In our framework, the local dataset is augmented by the GAN-based synthetic samples, which alleviates the non-IID problem. Furthermore, the PS in the traditional FL system only performs model aggregation based on the uploaded local models. In contrast, our framework generates high-confidence synthetic data to update and improve the global model at the PS. Overall, with a shared synthetic dataset and an effective mechanism for pseudo label update, SDA-FL can overcome the heterogeneous data distributions among clients and enhance the model update at the server. We envision that this framework can be applied to develop other data augmentation-based methods for more efficient federated learning.
sections.heading: 5 EXPERIMENTS
sections.text: In this section, we evaluate the proposed SDA-FL framework in the presence of non-IID data for both supervised learning and semi-supervised learning. The experimental results on different benchmark datasets demonstrate the effectiveness of the proposed framework compared with baseline methods. Besides, ablation studies are conducted to illustrate the impacts of key factors on the generated samples and the model convergence.
sections.heading: 5.1 EXPERIMENTAL SETUP
sections.text: Datasets We implement the SDA-FL framework in the scenario where all the real data are stored in clients and the PS only has access to the synthetic data. We use four benchmark datasets, including MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), Cifar-10 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011) to test the proposed method. The details about the datasets, models, and experimental settings are deferred to Appendix A. To demonstrate the effectiveness of our framework in handling different non-IID scenarios, we provide extensive experiments in the following five tasks.
  Tasks 1) Federated supervised learning: We compare our framework with FedAvg (McMahan et al., 2017), Fedprox (Li et al., 2020), Scaffold (Karimireddy et al., 2020), Naivemix, and Fedmix (Yoon et al., 2021) on MNIST, FashionMNIST, Cifar-10, and SVHN datasets. 2) Federated semi-supervised learning: We extend our framework to the semi-supervised learning setting in MNIST, FashionMNIST, and Cifar-10 datasets by performing pseudo labeling for the unlabeled data. To show the effectiveness of our SDA-FL method, we compare it with Local Fixmatch+Mixup (Diao et al., 2021), Local Fixmatch (Sohn et al., 2020), and Local Mixup (Zhang et al., 2018). 3) The trade-off between the privacy budget and performance: To investigate the influence of the differential privacy mechanism on the quality of the synthetic data and the model convergence, we validate our framework under different privacy budgets. In particular, we select the Fréchet Inception Distance (FID) (Heusel et al., 2017) as a metric to quantify the quality (similarity) of the synthetic data. 4) The effectiveness of updating pseudo labels and the server update: Compared with the traditional federated learning, SDA-FL leverages the pseudo labeling mechanism to generate highconfidence pseudo labels for the synthetic data, which are also used by the PS to further update the global model. To demonstrate the efficacy of these two mechanisms, we conduct ablation studies on a varying number of rounds for updating the pseudo labels and a varying number of server update steps. 5) Ablation studies of the computational cost and the number of samples for training the generators: To evaluate the performance of the generative models, we conduct ablation studies to investigate the influence of the training epochs and the number of samples on the quality (similarity) of the synthetic data.
sections.heading: 5.2 PERFORMANCE OF SDA-FL ON NON-IID FEDERATED LEARNING
sections.text: 1) Federated Supervised Learning We compare the performance of various methods under the same federated setting. With varying numbers of classes per client, the experimental results in Table 1 show that our framework outperforms other methods by a large margin, which is attributed to the GAN-based data augmentation that mitigates the detrimental effects of the data heterogeneity. Under the severe non-IID scenario (i.e., each client only has one class of data), our SDA-FL method maintains an accuracy of 88.46% in the SVHN classification task, while other baselines suffer from great performance degradation. In the Cifar-10 experiments, our framework outperforms the classic Naivemix and Fedmix algorithms at least by 5.0% with three classes of data at each client. This indicates that augmentation via synthetic data is more effective than averaging real data samples, although Fedmix adds another derivative term when averaging real images to explicitly handle the non-IID problem.
  Recently, GANs have been widely used to solve the non-IID and data scarce issues based on the federated learning in the medical field (Nguyen et al., 2021; Zhang et al., 2021), which can afford the additional computational cost for training the GANs. To further show the feasibility of SDA-FL, we include another comparison on a Covid-19 dataset (Chowdhury et al., 2020). See more details about the Covid-19 experiment in Appendix B.
  2) Federated Semi-Supervised Learning The results in Fig. 2 show that our framework achieves faster convergence and better performance than other algorithms, indicating its robustness and generalizability. Particularly, compared with Local Fixmatch+Local Mixup, our method improves the accuracy by nearly 10% in the FashionMNIST classification task. In the Cifar-10 dataset, the baseline methods are not able to train a usable global model (with a test accuracy below 40%), while our framework converges in this challenging scenario and improves the test accuracy by a significant margin. This is because the proposed pseudo labeling mechanism can provide high-quality labels for the synthetic and unlabeled samples, which benefits the federated training process.
  3) The trade-off between the privacy budget and performance The privacy budget provides the privacy protection for the generation of the synthetic dataset, but a small value of affects the quality of the synthetic samples and thus degrades the training performance. To investigate the impact of the privacy budget, we evaluate the performance of the proposed method under different
  values of . As shown in Fig.3, the proposed framework suffers from performance degradation with a small privacy budget. Compared with a protection-free scenario ( = ∞), setting a strict privacy budget ( = 5) (Xie et al., 2018) leads to around 0.61% and 2.59% accuracy drop in the MNIST and FashionMNIST datasets, respectively. Besides, we utilize the FID to measure the quality of the generated samples, where a smaller FID score denotes a better image quality. As illustrated in Fig.3, decreasing the privacy budget increases the FID score, which implies the degradation of the quality of the generated samples. Note that although our SDA-FL method is trained under the strict privacy requirement, compared with the results in Table 1, our framework still maintains desired performance over other methods.
  In the Cifar-10 classification task, as the generators are sensitive to the Gaussian noise added to the gradient, we adopt a large privacy budget ( = 50) when training the proposed method. Fig.3(c) shows that our framework outperforms Fedmix and Fedprox in federated supervised learning. To illustrate the privacy guarantee on the generators in our framework, we show some synthetic samples in Fig.4(a), which are extracted from the generators trained with = 200. In comparison with the samples from the private datasets, the synthetic images are blurry and noisy, which means the GAN models are not memorizing the real data, and the local privacy is not undermined. Besides, the performance improvements shown in Fig.3(c) and Table 1 indicate that the synthetic images with low perceptual quality can capture the necessary information for the model training.
  4) The effectiveness of updating pseudo labels and the server update In comparison to the traditional federated learning, our framework can update the global network with the synthetic data at the server, which has the potential to further improve the performance. The results in Fig.5 show that the model performance on FashionMNIST will drop nearly 3% without any server update. However, too many server updates also degrade the performance because too much involvement of the synthetic data will relieve the effects of the local update. The empirical results show that the model trained solely with the synthetic data (the∞ steps) can only get 66.0% accuracy, which highlights the necessity to effectively utilize the synthetic data as well as the local data for model training.
  Besides, keeping updating pseudo labels for the synthetic data improves the model performance. As illustrated in Fig.6, the accuracy gradually increases when continuously updating the pseudo labels,
  which demonstrates that our framework can improve the confidence level of the pseudo labels in the training process. Note that since our framework only updates the pseudo labels instead of the samples, the extra communication overhead is neglectable.
  5) The ablation studies of computational cost and the number of samples for training the generators We present the results in Fig.7 and Fig.8, which show that insufficient training rounds and training samples for generators result in low-quality synthetic data and degraded performance. The generator only gets an FID score of 217.81 (23.17 for real data) and an accuracy of 83.76% when training the generators with 30 rounds. In addition, as shown in Fig.8, under the fixed 30- round computational cost, using fewer samples for training the generators reduces the quality of the synthetic data and thus affects the performance. Nonetheless, despite the low computational cost (30 rounds) and the small number of samples (1000 samples) used to train the generators individually, our framework still outperforms other baselines (82.41% in Fedmix), demonstrating the robustness of the proposed method.
  In addition to the influence of the quality of synthetic data on the performance, other factors, such as the threshold τ , λ2, and the size of the synthetic dataset, also affect the global model. More ablation studies about these factors are shown in Appendix C.
sections.heading: 6 CONCLUSIONS
sections.text: We proposed a new data augmentation method to resolve the heterogeneous data distribution problem in federated learning, by introducing pretrained GANs to construct a differentially private global shared dataset. To effectively utilize the synthetic data, a novel framework, named Synthetic Data Aided Federated Learning (SDA-FL), was developed, which generates and updates confident pseudo labels for the synthetic data samples. By supplementing the non-IID private datasets with highquality synthetic data samples, this framework effectively improves the consistency among local models. Experiment results showed that SDA-FL outperforms many existing baselines by remarkable margins in various image classification tasks and is still functional under strict differential privacy protection.
sections.heading: A EXPERIMENTAL DETAIL
sections.text: Datasets To demonstrate the generalizability of our framework, we use five widely used datasets to test our frameworks, where the details of the datasets are summarized in Table 2. In all the experiments, we equally divide the training samples and assign them to the clients based on the nonIID setting. Specifically, given the number of classes per client as C and with K clients, the whole dataset will be split into K ∗C subsets, and each subset only has a single class. Then all the subsets of data will be randomly shuffled and distributed to clients. For example, in the MNIST experiment, under the scenario that each client only accesses two classes of data, given K = 10 clients, each
  client keeps 60000/10 = 6000 samples, each class with 3000 samples. The data distribution is kept the same in the comparison between our framework and other baselines. Except for the Covid-19 dataset, we set 10 clients for all the experiments, and all the clients participate in the training at every communication round.
  Models For fair comparisons, we adopt a simple CNN model that consists of two convolution layers and two fully-connected layers for the MNIST and FashionMNIST classification tasks. Meanwhile, ResNet18 (He et al., 2016) is used for classifying the Cifar-10, SVHN, and Covid-19 image datasets. To generate the qualified synthetic samples, we set a generator with four deconvolution layers and a discriminator with four layers convolution layers followed by a fully-connected layer.
  Experimental settings of tasks 1) Federated supervised learning: In the federated supervised and semi-supervised experiments, the parameters always are kept the same in our framework, and they are shown in the Table 3. In the baseline experiments, we set µ = 0.001, 0.01, 0.1, 1.0 for Fedprox, and λ = 0.01, 0.1, 0.2, 0.5 for Naivemix as well as Fedmix. The best performances of them are reported in the Table 1. 2) Federated semi-supervised learning: In this experiment, we compare our framework with other baselines in MNIST, FashionMNIST, and Cifar-10 datasets. Every client only keeps two classes of data, containing 50 labeled instances, and the rest are unlabeled. For fair comparisons, the total gradients steps of SDA-FL are the same as other methods. 3) The trade-off between the privacy budget and performance: We evaluate SDA-FL in MNIST and FashionMNIST with the privacy budget = 5, 10, 20,∞ and 30 training rounds (each round with 90 steps) for GANs. In the Cifar-10 experiment, we set the privacy budget = 50, 100, 200,∞ to train the GANs with 400 training rounds (each round with 90 steps). Every client keeps two classes of data in all experiments. 4) The effectiveness of updating pseudo labels and the server update: In this task, we test our
  framework in the FashionMNIST dataset and distribute two classes of data to every client. In our framework, the PS updates the pseudo labels for the synthetic dataset every communication step to maintain the confidence of the synthetic data. Compared with the federated supervised learning, we change to update the pseudo labels in the first 2, 10, 20 communication steps instead of updating the pseudo labels at all the communication steps in our framework. In the ablation study of the server update, we perform the experiments with different steps of server update (0, 20, 50, 100,∞) and batch size B = 64. 5) The ablation studies of computational cost and the number of samples for training the generators: The settings of all the ablation studies are the same as the federated supervised learning, and we test them in FashionMNIST with 2 classes of data at each client. All the experimental settings of ablation studies are shown in Table 4.
sections.heading: B THE APPLICATION IN THE COVID-19 DATASET
sections.text: Because of the scarcity of Pneumonia data, we only set up six clients in this experiment, each of which keeps two classes of data, and the data distribution is shown in Table 5. We test SDA-FL with 30 local steps, 10 server steps, and 50 rounds for training the generators locally. In addition to the baselines in the federated supervised learning task, we also add FedDPGAN (Zhang et al., 2021) for comparison, which trains a global GAN to solve the non-IID issue for medical applications, and the results are shown in Table 6.
  Our framework outperforms the other baseline algorithms by a remarkable margin. Particularly, SDA-FL is superior to FedDPGAN by 1.68% in the test accuracy, which indicates that the GANs which are trained individually generate higher-quality synthetic data than the global GAN trained based on the federated learning framework. Furthermore, in addition to addressing the non-IID issue, we also claim that the synthetic data can augment the data distribution, as SDA-FL is superior to FedAvg (IID) by 1.14% in the test accuracy. As a result, we assert that SDA-FL is able to show its advantages of solving non-IID problems and generating high-value data in medical applications.
sections.heading: C ABLATION STUDIES
sections.text: The size of the synthetic dataset Before the federated learning, all the clients constitute the synthetic dataset together. Larger synthetic datasets can typically provide more qualified data to clients and PS, thus improving performance, but they require larger storage space for clients. The results in Fig.9 show that, although the test accuracy on FashionMNIST decreases as the size of the synthetic dataset declines, our framework achieves 85.44% test accuracy with each client only uploading 200 samples, which is acceptable in exchange for less storage space requirement.
  The hyper-parameter λ2 λ2 is the parameter to control the use of the local real data. The results in Fig.10 show that compared with the normal settings of λ2, the test accuracy on FashionMNIST drops by nearly 5.0% with λ2 = 0.25, indicating that the training of the models is heavily reliant on the real data. Besides, an excessive λ2 also degrades the performance since it relieves the effects of the synthetic data and cannot mitigate the non-IID problem as much as possible. In comparison, a moderate λ2 is capable of combining the real data and the synthetic data intelligently and thus enhancing the models and solving the non-IID issue.
  Threshold τ Threshold τ sets the criterion for selecting the high-confidence pseudo labels. The results on FashionMNIST are shown in Fig.11, which indicates that a small threshold impairs the performance because some synthetic data with low-confidence pseudo labels are still considered qualified and used to train the models. An excessive threshold, on the other hand, filters out a lot of qualified data, which affects performance. Therefore, τ = 0.95 is a reasonable value to achieve a balance between quality and the quantity of the pseudo labels.