summaries: This paper introduces the self-supervised learning (SSL) framework for FL. Different SSL methods are investigated to study their feasibility under the FL setting. With the popular SimSiam framework, personalized federated SSL is proposed. The performance comparison indicates that representation regularization-based personalization method is able to outperform other variants. The authors merge self-supervised learning in personalized federated learning to solve the limited label and data heterogeneity problems in the local clients. They test several current algorithms under their framework. Then they propose an algorithm named Per- SSFL considering the balance between the consensus and personalization. Finally, they provide experiment results to support their claims and comprehensive analysis of what they find. The main contribution is to design a self-supervised FL framework with supportive experiment results. They also provide suggestions to choose appropriate algorithms and hyper-parameters under different settings. The submission is not anonymized, the author's name chaoyanghe appears frequently in the attached SSFL_Sumpplementary/SSFL-Source-Code. Therefore, it should be directly desk rejected.  --- Logging error --- Traceback (most recent call last):   File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 1025, in emit     msg = self.format(record)   File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 869, in format     return fmt.format(record)   File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 608, in format     record.message = record.getMessage()   File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 369, in getMessage     msg = msg % self.args This work is concerned with a very practical scenario of Federated Learning where the participating agents may not have access to labelled data. SimSiam architecture to learn useful feature representations with extensions that incorporate personalisation for local client models. The learnt representation are evaluated against a KNN classifier for analysing their usefulness. 
abstractText: Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy, reduce data migration costs, and break regulatory restrictions. An essential, but rarely studied, challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL, compared to centralized training, due to the fact that FL users are often reluctant to label their private data and edge devices do not provide an ideal interface to assist with annotation. Addressing label deficiency is also further complicated in FL, due to the heterogeneous nature of the data at edge devices and the need for developing personalized models for each user. We propose a self-supervised and personalized federated learning framework, named (SSFL), and a series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL framework, we analyze the compatibility of various centralized self-supervised learning methods in FL setting and demonstrate that SimSiam networks performs the best with the standard FedAvg algorithm. Moreover, to address the data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning including perFedAvg, Ditto, and local fine-tuning, among others. We further propose a novel personalized federated self-supervised learning algorithm, Per-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, we also develop a distributed training system and related evaluation protocol for SSFL. Using this training system, we conduct experiments on a synthetic non-I.I.D. dataset based on CIFAR-10, and an intrinsically non-I.I.D. dataset GLD-23K. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates that representation regularization-based personalization method is able to outperform other variants. Ablation studies on SSFL are also conducted to understand the role of batch size, non-I.I.D.ness, and the evaluation protocol.
authors.name: PERSONALIZED SELF-SUPERVISION
references.authors: H. Brendan McMahan Eider Moore Daniel Ramage Seth Hampson Blaise Agüera y Arcas
references.title: Communication-Efficient Learning of Deep Networks from Decentralized Data
references.venue: arXiv e-prints, art
references.authors: Jane Bromley James W Bentz Léon Bottou Isabelle Guyon Yann LeCun Cliff Moore Eduard Säckinger Roopak Shah
references.title: Signature verification using a “siamese” time delay neural network
references.venue: International Journal of Pattern Recognition and Artificial Intelligence,
references.authors: Han Cai Chuang Gan Ligeng Zhu Song Han
references.title: Tinytl: Reduce memory, not parameters for efficient on-device learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin
references.title: Unsupervised learning of visual features by contrasting cluster assignments
references.venue: arXiv preprint arXiv:2006.09882,
references.authors: Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin
references.title: Unsupervised learning of visual features by contrasting cluster
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: In International conference on machine learning,
references.authors: Xinlei Chen Kaiming He
references.title: Exploring simple siamese representation learning
references.authors: Marco Cuturi
references.title: Sinkhorn distances: Lightspeed computation of optimal transportation distances
references.authors: Alireza Fallah Aryan Mokhtari Asuman Ozdaglar
references.title: Personalized federated learning: A metalearning approach
references.venue: arXiv preprint arXiv:2002.07948,
references.authors: Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre H Richemond Elena Buchatskaya Carl Doersch Bernardo Avila Pires Zhaohan Daniel Guo Mohammad Gheshlaghi Azar
references.title: Bootstrap your own latent: A new approach to self-supervised learning
references.venue: arXiv preprint arXiv:2006.07733,
references.authors: Chaoyang He Songze Li Jinhyun So Mi Zhang Hongyi Wang Xiaoyang Wang Praneeth Vepakomma Abhishek Singh Hang Qiu Li Shen Peilin Zhao Yan Kang Yang Liu Ramesh Raskar Qiang Yang Murali Annavaram Salman Avestimehr
references.title: Fedml: A research library and benchmark for federated machine learning
references.venue: arXiv preprint arXiv:2007.13518,
references.authors: Andrew G Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto Hartwig Adam
references.title: Mobilenets: Efficient convolutional neural networks for mobile vision applications
references.venue: arXiv preprint arXiv:1704.04861,
references.authors: Tzu-Ming Harry Hsu Hang Qi Matthew Brown
references.title: Measuring the effects of non-identical data distribution for federated visual classification
references.authors: Sohei Itahara Takayuki Nishio Yusuke Koda Masahiro Morikura Koji Yamamoto
references.title: Distillationbased semi-supervised federated learning for communication-efficient collaborative training with non-iid private data
references.venue: arXiv preprint arXiv:2008.06180,
references.authors: Wonyong Jeong Jaehong Yoon Eunho Yang Sung Ju Hwang
references.title: Federated semi-supervised learning with inter-client consistency
references.venue: arXiv preprint arXiv:2006.12097,
references.authors: far O. Koyejo Tancrède Lepoint Yang Liu Prateek Mittal M. Mohri R. Nock A. Özgür R. Pagh Mariana Raykova Hang Qi D. Ramage R. Raskar D. Song Weikang Song Sebastian U. Stich Ziteng Sun A.T. Suresh Florian Tramèr Praneeth Vepakomma Jianyu Wang Li Xiong Zheng Xu Qiang Yang F. Yu Han Yu Sen Zhao
references.title: Advances and open problems in federated learning
references.venue: Found. Trends Mach. Learn.,
references.authors: Peter Kairouz H Brendan McMahan Brendan Avent Aurélien Bellet Mehdi Bennis Arjun Nitin Bhagoji Keith Bonawitz Zachary Charles Graham Cormode Rachel Cummings
references.title: Advances and open problems in federated learning
references.venue: arXiv preprint arXiv:1912.04977,
references.authors: Dong-Hyun Lee
references.title: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks
references.authors: Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar Virginia Smith
references.title: Federated optimization in heterogeneous networks
references.venue: arXiv preprint arXiv:1812.06127,
references.authors: Tian Li Shengyuan Hu Ahmad Beirami Virginia Smith
references.title: Ditto: Fair and robust federated learning through personalization
references.authors: Xinle Liang Yang Liu Jiahuan Luo Yuanqin He Tianjian Chen Qiang Yang
references.title: Self-supervised cross-silo federated neural architecture search
references.venue: arXiv preprint arXiv:2101.11896,
references.authors: Yi Liu Xingliang Yuan Ruihui Zhao Yifeng Zheng Yefeng Zheng
references.title: Rc-ssfl: Towards robust and communication-efficient semi-supervised federated learning system
references.venue: arXiv preprint arXiv:2012.04432,
references.authors: Zewei Long Liwei Che Yaqing Wang Muchao Ye Junyu Luo Jinze Wu Houping Xiao Fenglong Ma
references.title: Fedsemi: An adaptive federated semi-supervised learning framework
references.authors: Brendan McMahan Eider Moore Daniel Ramage Seth Hampson Blaise Aguera y Arcas
references.title: Communication-efficient learning of deep networks from decentralized data
references.venue: In Artificial Intelligence and Statistics,
references.authors: Takeru Miyato Shin-Ichi Maeda Masanori Koyama Shin Ishii
references.title: Virtual adversarial training: A regularization method for supervised and semi-supervised learning
references.authors: Takeru Miyato Shin-ichi Maeda Masanori Koyama Shin Ishii
references.title: Virtual adversarial training: a regularization method for supervised and semi-supervised learning
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Sashank Reddi Zachary Charles Manzil Zaheer Zachary Garrett Keith Rush Jakub Konečnỳ Sanjiv Kumar H Brendan McMahan
references.title: Adaptive federated optimization
references.venue: arXiv preprint arXiv:2003.00295,
references.authors: Aaqib Saeed Flora D Salim Tanir Ozcelebi Johan Lukkien
references.title: Federated self-supervised learning of multisensor representations for embedded intelligence
references.venue: IEEE Internet of Things Journal,
references.authors: Tim Salimans Diederik P. Kingma
references.title: Weight normalization: A simple reparameterization to accelerate training of deep neural networks
references.venue: CoRR, abs/1602.07868,
references.authors: Mingxing Tan Quoc Le
references.title: Efficientnet: Rethinking model scaling for convolutional neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Jianyu Wang Qinghua Liu Hao Liang Gauri Joshi H Vincent Poor
references.title: Tackling the objective inconsistency problem in heterogeneous federated optimization
references.venue: arXiv preprint arXiv:2007.07481,
references.authors: Hang Qi Sashank J. Reddi Peter Richtárik K. Singhal Virginia Smith M. Soltanolkotabi Weikang Song A.T. Suresh Sebastian U. Stich Ameet S. Talwalkar Hongyi Wang Blake E. Woodworth Shanshan Wu Felix X. Yu Honglin Yuan M. Zaheer Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu
references.title: A field guide to federated optimization
references.venue: ArXiv, abs/2107.06917,
references.authors: Zhirong Wu Yuanjun Xiong Stella Yu Dahua Lin
references.title: Unsupervised feature learning via nonparametric instance-level discrimination, 2018
references.authors: Fengda Zhang Kun Kuang Zhaoyang You Tao Shen Jun Xiao Yin Zhang Chao Wu Yueting Zhuang Xiaolin Li
references.title: Federated unsupervised representation learning
references.venue: arXiv preprint arXiv:2010.08982,
references.authors: Zhengming Zhang Yaoqing Yang Zhewei Yao Yujun Yan Joseph E Gonzalez Michael W Mahoney
references.title: Improving semi-supervised federated learning by reducing the gradient diversity of models
references.venue: arXiv preprint arXiv:2008.11364,
references.authors: Yuchen Zhao Hanyang Liu Honglin Li Payam Barnaghi Hamed Haddadi
references.title: Semi-supervised federated learning for activity recognition
references.venue: arXiv preprint arXiv:2011.00851,
sections.text: Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy, reduce data migration costs, and break regulatory restrictions. An essential, but rarely studied, challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL, compared to centralized training, due to the fact that FL users are often reluctant to label their private data and edge devices do not provide an ideal interface to assist with annotation. Addressing label deficiency is also further complicated in FL, due to the heterogeneous nature of the data at edge devices and the need for developing personalized models for each user. We propose a self-supervised and personalized federated learning framework, named (SSFL), and a series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL framework, we analyze the compatibility of various centralized self-supervised learning methods in FL setting and demonstrate that SimSiam networks performs the best with the standard FedAvg algorithm. Moreover, to address the data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning including perFedAvg, Ditto, and local fine-tuning, among others. We further propose a novel personalized federated self-supervised learning algorithm, Per-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, we also develop a distributed training system and related evaluation protocol for SSFL. Using this training system, we conduct experiments on a synthetic non-I.I.D. dataset based on CIFAR-10, and an intrinsically non-I.I.D. dataset GLD-23K. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates that representation regularization-based personalization method is able to outperform other variants. Ablation studies on SSFL are also conducted to understand the role of batch size, non-I.I.D.ness, and the evaluation protocol.
sections.heading: 1 INTRODUCTION
sections.text: Federated Learning (FL) is a contemporary distributed machine learning paradigm that aims at strengthening data privacy, reducing data migration costs, and breaking regulatory restrictions (Kairouz et al., 2021; Wang et al., 2021). It has been widely applied to computer vision, natural language processing, and data mining. However, there are two main challenges impeding its wider adoption in machine learning. One is data heterogeneity, which is a natural property of FL in which diverse clients may generate datasets with different distributions due to behavior preferences (e.g., the most common cause of heterogeneity is skewed label distribution which might result from instances where some smartphone users take more landscape pictures, while others take more photos of daily life). The second challenge is label deficiency at the edge, which is relatively less studied. This issue is more severe at the edge than in a centralized setting because users are reluctant to annotate their private and sensitive data, and/or smartphones and IoT devices do not have a user-friendly interface to assist with annotation.
  To mitigate the data heterogeneity issue among clients, researchers have proposed algorithms for training a global model FedAvg (McMahan et al., 2017), FedProx (Li et al., 2018), FedNova (Wang et al., 2020), FedOPT (Reddi et al., 2020), as well as personalized FL frameworks (e.g., pFedMe, Ditto, Per-FedAvg). These algorithms all depend on the strong assumption that the data at the edge has sufficient labels. To address the label deficiency issue in FL, recent works (Liu et al., 2020; Long et al., 2020; Itahara et al., 2020; Jeong et al., 2020; Liang et al., 2021; Zhao et al., 2020; Zhang et al., 2020a;b) assume that the server or client has a fraction of labeled data and use semi-supervised methods such as consistency loss (Miyato et al., 2018) or pseudo labeling (Lee, 2013) to train a global model. A more realistic but challenging setting is fully unsupervised training. Although a recent work in FL (Saeed et al., 2020) attempts to address this challenge through Siamese networks proposed around thirty years ago (Bromley et al., 1993), its design does not tackle data heterogeneity for learning personalized models, and it only trains on small-scale sensor data in IoT devices. Moreover, these existing works in FL have not examined recent progress in the Self-Supervised Learning (SSL) community where methods such as SimCLR (Chen et al., 2020), SwAV(Caron et al., 2021), BYOL (Grill et al., 2020), and SimSiam (Chen & He, 2020) have shown tremendous improvement in reducing the amount of labeled data required to achieve state-of-the-art performance. As such, it remains still unclear how these SSL methods can be incorporated into FL and how well they would perform, especially when intertwined with the data heterogeneity challenge that does not exist in centralized training.
  In this paper, we propose Self-Supervised Federated Learning (SSFL), a unified self-supervised and personalized federated learning framework, and a series of algorithms under this framework to address these challenges. As shown in Figure 1, this framework brings state-of-the-art self-supervised learning algorithms to the realm of FL in order to enable training without using any supervision, while also integrating model personalization to deal with data heterogeneity (Section 3.1). More specifically, under the SSFL framework, we analyze the compatibility of various centralized selfsupervised learning methods in the FL setting and demonstrate that SimSiam networks performs the best with the standard FedAvg algorithm (Section 3.2). Moreover, to address the data heterogeneity at edge devices, we have innovated a series of algorithms that broaden the reach of existing supervised personalization algorithms into the setting of self-supervised learning, including perFedAvg (Fallah et al., 2020), Ditto (Li et al., 2021), and local fine-tuning, among others. We further propose a novel personalized federated self-supervised learning algorithm, per-SSFL (Section 3.3), which balances personalization and consensus by carefully regulating the distance between the local and global representations of data (shown as the yellow block in Figure 1).
  To provide a comprehensive and comparative analysis of the proposed algorithms, we also develop a distributed training system and evaluation protocol for SSFL. Using this training system, we conduct experiments on a synthetic non-I.I.D. dataset based on CIFAR-10 and a natural non-I.I.D. dataset GLD-23K. Our experimental results demonstrate that all algorithms in our framework work reliably. In FL, the gap of evaluation accuracy between supervised learning and unsupervised learning is small. Personalized SSFL performs better than FedAvg-based SSFL. We also conduct ablation studies to fully understand the SSFL framework, namely the role of batch size, different degrees of non-I.I.D.ness, and performance in more datasets. Finally, our unified API design can serve as a suitable platform and baseline, enabling further developments of more advanced SSFL algorithms.
sections.heading: 2 PRELIMINARIES
sections.text: SSFL builds upon two fundamental areas in machine learning: federated optimization and selfsupervised learning. Thus, we first introduce some basics and formulations in these areas.
sections.heading: 2.1 FEDERATED OPTIMIZATION
sections.text: Federated optimization refers to the distributed optimization paradigm that a network of K devices collaboratively solve a machine learning task. In general, it can be formulated as a distributed optimization problem with the form (McMahan et al., 2017): minθ ∑K k=1 |Dk| |D| L(θ,Dk). Here, each device k has a local dataset Dk drawn from a local distribution Xk. The combined dataset D = ∪Kk=1Dk is the union of all local datasets Dk. θ represents the model weight of a client model. L is the client’s local loss function that measures the local empirical risk over the heterogeneous dataset Dk. Under this formulation, to mitigate the non-I.I.D. issue, researchers have proposed algorithms such as FedProx (Li et al., 2018), FedNova (Wang et al., 2020), and FedOPT (Reddi et al., 2020) for training a global model, as well as personalized FL frameworks such as Ditto (Li et al., 2021), and Per-FedAvg (Fallah et al.). All of these algorithms have a strong assumption that data at the edge have sufficient labels, meaning that their analysis and experimental settings are based on a supervised loss function, such as the cross-entropy loss for image classification.
sections.heading: 2.2 SELF-SUPERVISED LEARNING
sections.text: Self-supervised learning (SSL) aims to learn meaningful representations of samples without human supervision. Formally, it aims to learn an encoder function fθ : X 7→ Rd where θ is the parameter of the function, X is the unlabeled sample space (e.g. image, text), and the output is a d dimensional vector containing enough information for downstream tasks such as image classification and segmentation. The key to SSL’s recent success is the inductive bias that ensures a good representation encoder remains consistent under different perturbations of the input (i.e. consistency regularization).
  One prominent example among recent advances in modern SSL frameworks is the Siamese architecture (Bromley et al., 1993) and its improved variants SimCLR (Chen et al., 2020), SwAV (Caron et al., 2021), BYOL (Grill et al., 2020), and SimSiam (Chen & He, 2020). Here we review the most elegant architecture, SimSiam, and defer the description and comparison of the other three to Appendix A. SimSiam proposes a two-head architecture in which two different views (augmentations) of the same image are encoded by the same network fθ. Subsequently, a predictor Multi Layer Perceptron (MLP) hθ and a stop-gradient operation denoted by ·̂ are applied to both heads. In the SSL context, “stop gradient” means that the optimizer stops at a specific neural network layer during the back propagation and views the parameters in preceding layers as constants. Here, θ is the concatenation of the parameters of the encoder network and the predictor MLP. The algorithm aims to minimize the negative cosine similarity D(·, ·) between two heads. More concretely, the loss is defined as
  LSS(θ,D) = 1 |D| ∑ x∈D D(fθ(T (x)), ¤ hθ(fθ(T (x)))), (1)
  where T represents stochastic data augmentation and D is the data set.
sections.heading: 3 SSFL: SELF-SUPERVISED FEDERATED LEARNING
sections.text: In this section, we propose SSFL, a unified framework for self-supervised federated learning. Specifically, we introduce the method by which SSFL works for collaborative training of a global model and personalized models, respectively.
sections.heading: 3.1 GENERAL FORMULATION
sections.text: We formulate self-supervised federated learning as the following distributed optimization problem:
  min Θ
  {θk}k∈[K]
  G (L (θ1,Θ;X1) , . . . ,L (θK ,Θ;XK)) (2)
  where θk is the parameter for the local model (fθk , hθk); Θ is the parameter for the global model (fΘ, hΘ); L(θk,Θ;Xk) is a loss measuring the quality of representations encoded by fθk and fΘ on the local distribution Xk; and G(·) denotes the aggregation function (e.g. sum of client losses weighted by |Dk||D| ). To capture the two key challenges in federated learning (data heterogeneity and label deficiency), we hold two core assumptions in the proposed framework: (1) Xk of all clients are heterogeneous (non-I.I.D.), and (2) there is no label.
  To tackle the above problem, we propose a unified training framework for federated self-supervised learning, as described in Algorithm 1. This framework can handle both non-personalized and personalized federated training. In particular, if one enforces the constraint θk = Θ for all clients k ∈ [K], the problem reduces to learning a global model. When this constraint is not enforced, θk can be different for each client, allowing for model personalization. ClientSSLOPT is the local optimizer at the client side which solves the local sub-problem in a self-supervised manner. ServerOPT takes the update from the client side and generates a new global model for all clients.
  Algorithm 1: SSFL: A Unified Framework for Self-supervised Federated Learning input :K,T,Θ(0), {θ(0)k }k∈[K], CLIENTSSLOPT , SERVEROPT
  1 for t = 0, . . . , T − 1 do 2 Server randomly selects a subset of devices S(t) 3 Server sends the current global model Θ(t) to S(t) 4 for device k ∈ S(t) in parallel do 5 Solve local sub-problem of equation 2:
  θk,Θ (t) k ← CLIENTSSLOPT (θ (t) k ,Θ (t),∇L (θk,Θ;Xk))
  6 Send ∆(t)k := Θ (t) k −Θ (t) back to server 7 Θ(t+1) ← SERVEROPT Ä Θ(t), {∆(t)k }k∈S(t) ä return :{θk}k∈[K],Θ(T )
  Next, we will introduce specific forms of ClientSSLOPT and ServerOPT for global training and personalized training.
sections.heading: 3.2 GLOBAL-SSFL: COLLABORATION TOWARDS A GLOBAL MODEL WITHOUT SUPERVISION
sections.text: To train a global model using SSFL, we design a specific form of ClientSSLOPT using SimSiam. We choose SimSiam over other contemporary self-supervised learning frameworks (e.g., SimSiam, SwAV, BYOL) based on the following analysis as well as experimental results (see Section 5.1).
  The simplicity in neural architecture and training method. SimSiam’s architecture and training method are relatively simple. For instance, compared with SimCLR, SimSiam has a simpler loss function; compared with SwAV, SimSiam does not require an additional neural component (prototype vectors) and Sinkhorn-Knopp algorithm; compared with BYOL, SimSiam does not need to maintain an additional moving averaging network for an online network. Moreover, the required batch size of SimSiam is the smallest, making it relatively friendly for resource-constrained federated learning. A more comprehensive comparison can be found in Appendix A.
  Interpretability of SimSiam leads to simpler local optimization. More importantly, SimSiam is more interpretable from an optimization standpoint which simplifieds the local optimization. In particular, it can be viewed as an implementation of an Expectation-Maximization (EM) like algorithm, meaning that optimizing LSS in Equation 1 is implicitly optimizing the following objective
  min θ,η E T x∼X
  î ‖fθ(T (x))− ηx‖22 ó . (3)
  Here, fθ is the encoder neural network parameterized by θ. η is an extra set of parameters, whose size is proportional to the number of images, and ηx refers to using the image index of x to access a sub-vector of η. This formulation is w.r.t. both θ and η and can be optimized via an alternating algorithm. At time step t, the ηtx update takes the form η t x ← ET [fθt(T (x))], indicating that ηtx is assigned the average representation of x over the distribution of augmentation. However, it is impossible to compute this step by going over the entire dataset during training. Thus, SimSiam uses one-step optimization to approximate the EM-like two-step iteration by introducing the predictor hθ
  to approximate η and learn the expectation (i.e. hθ (z) ≈ ET [fθ(T (x))]) for any image x. After this approximation, the expectation ET [·] is ignored because the sampling of T is implicitly distributed across multiple epochs. Finally, we can obtain the self-supervised loss function in Equation 1, in which negative cosine similarity D is used in practice (the equivalent L2 distance is used in Equation 3 for the sake of analysis). Applying equation 1 as ClientSSLOPT simplifies the local optimization for each client in a self-supervised manner.
sections.heading: 3.3 PER-SSFL: LEARNING PERSONALIZED MODELS WITHOUT SUPERVISION
sections.text: In this section, we explain how SSFL addresses the data heterogeneity challenge by learning personalized models. Inspired by the interpretation in Section 3.2, we define the following sub-problem for each client k ∈ [K]:
  min θk,ηk E T x∼Xk
  ï ‖fθk(T (x))− ηk,x‖ 2 2 + λ
  2 ‖ηk,x −H∗x‖ 2 2 ò s.t. Θ∗,H∗ ∈ arg min
  Θ,H n∑ i=1 |Dk| |D| E T x∼Xi
  î ‖fΘ(T (x))−Hx‖22 ó (4) Compared to global training, we additionally include Θ, the global model parameter, andH, the global version of η, and the expected representations which correspond to the personalized parameters θk and ηk. In particular, through the term ‖ηk,x −H∗x‖ 2 2, we aim for the expected local representation of any image x to reside within a neighborhood around the expected global representation of x. Therefore, by controlling the radius of the neighborhood, hyperparameter λ helps to balance consensus and personalization.
  Algorithm 2: Per-SSFL input :K,T, λ,Θ(0), {θ(0)i }k∈[K], s: number of local iteration, β: learning rate
  1 for t = 0, . . . , T − 1 do 2 Server randomly selects a subset of devices S(t) 3 Server sends the current global model Θ(t) to S(t) 4 for device k ∈ S(t) in parallel do 5
  CLIENTSSLOPT
  Sample mini-batch Bk from local dataset Dk, and do s local iterations /* Optimize the global parameter Θ locally */
  6 Z1, Z2 ← fΘ(t)(T (Bk)), fΘ(t)(T (Bk)) 7 P1, P2 ← hΘ(t)(Z1), hΘ(t)(Z2) 8 Θ
  (t) k ← Θ (t) − β∇Θ(t) D(P1,Ẑ2)+D(P2,Ẑ1) 2 , where ·̂ stands for stop-gradient
  /* Optimize the local parameter θk */ 9 z1, z2 ← fθk (T (Bk)), fθk (T (Bk))
  10 p1, p2 ← hθk (z1), hθk (z2) 11 θk ← θk − β∇θk
  Ä D(p1,”z2)+D(p2,”z1)
  2 + λD(p1,P1)+D(p1,P2)+D(p2,P1)+D(p2,P2) 4 ä 12 Send ∆(t)k := Θ (t) k −Θ (t) back to server
  13 SERVEROPTΘ (t+1) ← Θ(t) + ∑ k∈S(t) |Dk| |D| ∆ (t) k
  return :{θi}i∈[n],Θ(T )
  We see that Equation 4 in the above objective is an optimization problem w.r.t. both θ and η. However, as the above target is intractable in practice, following an analysis similar to Section 3.2, we use the target below as a surrogate:
  min θk
  LSS (θk, Dk) + λ |Dk| ∑ x∈Dk D (hθk (fθk (T (x))) , hΘ∗ (fΘ∗ (T (x)))) (5)
  s.t. Θ∗ ∈ arg min Θ LSS (Θ, D) (6)
  In practice, Θ can be optimized independently of θk through the FedAvg (McMahan et al., 2017) algorithm. To make the computation more efficient, we also apply the symmetrization trick proposed in (Chen & He, 2020). We refer to this algorithm as Per-SSFL and provide a detailed description in Algorithm 2 (also illustrated in Fig. 1).
  Regarding the theoretical analysis. To our knowledge, all self-supervised learning frameworks do not have any theoretical analysis yet, particularly the SimSiam dual neural network architecture.
  Our formulation and optimization framework are interpretable, they are built based on an EM-like algorithm for SimSiam and minimizing the distance between the private model and the global model’s data representation.
  Innovating baselines to verify SSFL. Note that we have not found any related works that explore a Siamese-like SSL architecture in an FL setting. As such, to investigate the performance of our proposed algorithm, we further propose several other algorithms that can leverage the SSFL framework. 1. LA-SSFL. We apply FedAvg (McMahan et al., 2017) on the SimSiam loss LSS for each client to obtain a global model. We perform one step of SGD on the clients’ local data for local adaption; 2. MAML-SSFL. This algorithm is inspired by perFedAvg (Fallah et al.) and views the personalization on each devices as the inner loop of MAML (Finn et al.). It aims to learn an encoder that can be easily adapted to the clients’ local distribution. During inference, we perform one step of SGD on the global model for personalization; 3. BiLevel-SSFL. Inspired by Ditto (Li et al., 2021), we learn personalized encoders on each client by restricting the parameters of all personalized encoders to be close to a global encoder independently learned by weighted aggregation. More details of these algorithms, formulation, and pseudo code are introduced in Appendix B. In Section 5.3, we will show the comparison results for these proposed SSFL algorithmic variants.
sections.heading: 4 TRAINING SYSTEM AND EVALUATION PIPELINE FOR SSFL
sections.heading: A Distributed Training System to accelerate the algorithmic exploration in SSFL framework.
sections.text: We also contributed to reproducible research via our distributed training system. This is necessary for two reasons: (1) Running a stand-alone simulation (training client by client sequentially) like most existing FL works requires a prohibitively long training time when training a large number of clients. In SSFL, the model size (e.g., ResNet-18 v.s. shallow CNNs used in the original FedAvg paper) and the round number for convergence (e.g., 800 epochs in the centralized SimSiam framework) is relatively larger than in FL literature. By running all clients in parallel on multiple CPUs/GPUs, we can largely accelerate the process. (2) Given that SSFL is a unified and generic learning framework, researchers may develop more advanced ways to improve our work. As such, we believe it is necessary to design unified APIs and system frameworks in line with the algorithmic aspect of SSFL. See Appendix C for more details on our distributed training system.
  Evaluation Pipeline. In the training phase, we use a KNN classifier (Wu et al., 2018) as an online indicator to monitor the quality of the representations generated by the SimSiam encoder. For Global-SSFL, we report the KNN test accuracy using the global model and the global test data, while in Per-SSFL, we evaluate all clients’ local encoders separately with their local test data and report their averaged accuracy. After self-supervised training, to evaluate the performance of the trained encoder, we freeze the encoder and attach a linear classifier to the output of the encoder. For Global-SSFL, we can easily verify the performance of SimSiam encoder by training the attached linear classifier with FedAvg. However, for Per-SSFL, each client learns a personalized SimSiam encoder. As the representations encoded by personalized encoders might reside in different spaces, using a single linear classifier trained by FedAvg to evaluate these representations is unreasonable (see experiments in Section 5.4.3). As such, we suggest an additional evaluation step to provide a more representative evaluation of Per-SSFL’s performance: for each personalization encoder, we use the entire training data to train the linear classifier but evaluate on each client’s local test data.
sections.heading: 5 EXPERIMENTS
sections.text: In this section, we introduce experimental results for SSFL with and without personalization and present a performance analysis on a wide range of aspects including the role of batch size, different degrees of non-IIDness, and understanding the evaluation protocol.
  Implementation. We develop the SSFL training system to simplify and unify the algorithmic exploration. Details of the training system design can be found in Appendix C. We deploy the system in a distributed computing environment which has 8 NVIDIA A100-SXM4 GPUs with sufficient memory (40 GB/GPU) to explore different batch sizes (Section 5.4.1). Our training framework can run multiple parallel training workers in a single GPU, so it supports federated training with a large number of clients. The client number selected per round used in all experiments is 10, which is a reasonable setting suggested by recent literature (Reddi et al., 2020).
  Learning Task. Following SimCLR (Chen et al., 2020), SimSiam (Chen & He, 2020), BYOL (Grill et al., 2020), and SwAV (Caron et al., 2020) in the centralized setting, we evaluate SSL for the image classification task and use representative datasets for federated learning.
  Dataset. We run experiments on synthetic non-I.I.D. dataset CIFAR-10 and intrinsically non-I.I.D. dataset Google Landmark-23K (GLD-23K), which are suggested by multiple canonical works in the FL community (Reddi et al., 2020; He et al., 2020; Kairouz et al., 2019). For the non-I.I.D. setting, we distribute the dataset using a Dirichlet distribution (Hsu et al., 2019), which samples pc ∼ Dir(α) (we assume a uniform prior distribution) and allocates a pc,k proportion of the training samples of class c to local client k. We provide a visualization of the data distribution in Appendix E.1.
  Model Architecture. For the model architecture, ResNet-18 is used as the backbone of the SimSiam framework, and the predictor is the same as that in the original paper.
  Next, we focus on results from the curated CIFAR-10 dataset and defer GLD-23K to Appendix D.
sections.heading: 5.1 COMPARISONS ON SIMSIAM, SIMCLR, SWAV, AND BYOL
sections.text: Our first experiment determines which SSL method is ideal for FL settings. We run experiments using FedAvg for these four methods and obtain two findings: (1) SimSiam outperforms SimCLR in terms of accuracy; (2) BYOL and SwAV do not work in FL; we tested a wide range of hyper-parameters, but they still are unable to converge to a reasonable accuracy. These experimental results confirm our analysis in Section 3.2.
sections.heading: 5.2 EVALUATION ON GLOBAL-SSFL
sections.text: The goal of this experiment is to understand the accuracy gap between supervised and self-supervised federated learning in both I.I.D. and non-I.I.D. settings where we aim to train a global model from private data from clients.
  Setup and Hyper-parameters. We evaluate Global-SSFL using non-I.I.D. data from CIFAR-10: we set α = 0.1 for the Dirichlet distribution. For supervised learning, the test accuracy is evaluated on a global test dataset. For self-supervised training, we follow the evaluation protocol introduced in Section 4. We use SGD with Momentum as the client-side optimizer and a learning rate scheduler across communication rounds. We searched the learning rate on the grid of {0.1, 0.3, 0.01, 0.03} and report the best result. The experiment is run three times using the same learning rate with fixed random seeds to ensure reproducibility. The training lasts for 800 rounds, which is sufficient to achieve convergence for all methods. More hyperparameters are in Appendix E.2.
  We display the training curves in Figure 10 which demonstrates that SSFL can converge reliably in both I.I.D. and non-I.I.D. settings. For the I.I.D. data, we find that SSFL can achieve the same accuracy as the centralized accuracy report in the SimSiam paper (Chen & He, 2020). For the non-I.I.D. data, SSFL achieves a reasonable accuracy compared to the centralized accuracy. The accuracy comparisons in different dimensions (supervised v.s. self-supervised; I.I.D. v.s. non-I.I.D.) are summarized in Table 1.
sections.heading: 5.3 EVALUATION ON PER-SSFL
sections.text: Based on the results of SSFL with FedAvg, we further add the personalization components for SSFL introduced in Section 3.3 (Per-SSFL).
  Setup and Hyper-parameters. For a fair comparison, we evaluate Per-SSFL on non-I.I.D. data from CIFAR-10 and set α = 0.1 for the Dirichlet distribution. For Per-SSFL training, we follow the evaluation protocol introduced in Section 4. Similar to SSFL, we use SGD with Momentum as the client-side optimizer and the learning rate scheduler across communication rounds. We search for the learning rate on a grid of {0.1, 0.3, 0.01, 0.03} and report the best result. For Per-SSFL and BiLevel-SSFL, we also tune the λ of the regularization term with a search space {1, 10, 0.1, 0.01, 0.001}. The experiments are run three times with the same learning rate and with fixed random seeds to ensure reproducibility. The training also lasts for 800 communication rounds, which is the same as Global-SSFL. Other hyperparameters can be found in Appendix E.2.
  We illustrate our results in Figure 10 and Table 2. To confirm the convergence, we draw loss curves for all methods in Figure 10(b) (note that they have different scaled values due to the difference of their loss functions). Figure 10(b) indicates that Per-SSFL performs best among all methods. MAML-SSFL is also a suggested method since it obtains comparable accuracy. LA-SSFL is a practical method, but it does not perform well in the self-supervised setting. In Figure 10(b), the averaged personalized accuracy of LA-SSFL diverges in the latter phase. Based on BiLevel-SSFL’s result, we can conclude such a method is not a strong candidate for personalization, though it shares similar objective functions as Per-SSFL. This indicates that regularization through representations encoded by SimSiam outperforms regularization through weights.
sections.heading: 5.4 PERFORMANCE ANALYSIS
sections.text: 5.4.1 ROLE OF BATCH SIZE
  FL typically requires a small batch size to enable practical training on resource-constrained edge devices. Therefore, understanding the role of batch size in SSFL is essential to practical deployment and applicability. To investigate this, we use different batch sizes and tune the learning rate to find the optimal accuracy for each one. The results in Figure 4 show that SSFL requires a large batch size (256); otherwise, it reduces the accuracy or diverges during training. Since system efficiency is not the focus of this paper, we use gradient accumulation, which is a simple yet effective method. We fix the batch size at 32 and use accumulation step 8 for all experiments. For an even larger batch size (e.g., 512), the memory cost is significant, though there is no notable gain in accuracy. Therefore, we discontinue the search for batch sizes larger than larger than 256. A more advanced method includes batch-size-one training
  and knowledge distillation. We defer the discussion to Appendix F.
sections.heading: 5.4.2 ON DIFFERENT DEGREES OF NON-I.I.D.NESS
sections.text: We investigate the impact of the degree of data heterogeneity on the SSFL performance. We compare the performance between α = 0.1 and α = 0.5. These two settings provide a non-negligible gap
  in the label distribution in each client (see our visualization in Appendix E.1). Figure 5(a) and 5(b) shows the learning curve comparisons. It is clearly observed that the higher degree of data heterogeneity makes it converge more slowly, adversely affecting the accuracy.
  0 20 40 60 80 100
  round 0.3
  0.4
  0.5
  0.6
  0.7
  0.8
  Per-SSFL
  LA-SSFL
  Accuracy
  Figure 6: Understanding the Evaluation Protocol
sections.heading: 5.4.3 UNDERSTANDING THE LINEAR EVALUATION OF PERSONALIZED ENCODERS
sections.text: As we discussed in 4, in SSFL, we can easily verify the quality of the SimSiam encoder using federated linear evaluation; however, in Per-SSFL, each client learns a personalized SimSiam encoder. Such heterogeneity in diverse encoders makes a fair evaluation difficult. To demonstrate this, we run experiments with naive federated linear evaluation on personalized encoders and surprisingly find that such an evaluation protocol downgrades the performance. As shown in Figure 6, the federated linear evaluation for Per-SSFL performs worse than even LA-SSFL. This may be attributed to the fact that the naive aggregation drags close to the parameter space of all heterogeneous encoders, making the encoder degenerate in terms of personalization.
sections.heading: 6 RELATED WORKS
sections.text: Federated Learning (FL) with Personalization. pFedMe (Dinh et al.), perFedAvg (Fallah et al., 2020), and Ditto (Li et al., 2021) are some representative works in this direction. However, these methods all have a strong assumption that users can provide reliable annotations for their private and sensitive data, which we argue to be very unrealistic and impractical. Label deficiency in FL. There are a few related works to tackle label deficiency in FL (Liu et al., 2020; Long et al., 2020; Itahara et al., 2020; Jeong et al., 2020; Liang et al., 2021; Zhao et al., 2020; Zhang et al., 2020b). Compared to these works, our proposed SSFL does not use any labels during training. FedMatch (Jeong et al., 2020) and FedCA (Zhang et al., 2020a) requires additional communication costs to synchronize helper models or public labeled dataset. (Saeed et al., 2020) addresses the fully unsupervised challenge on small-scale sensor data in IoT devices. However, compared to our work, it uses the Siamese networks proposed around thirty years ago (Bromley et al., 1993), lacking consideration on the advance in the past two years (i.e., SimCLR (Chen et al., 2020), SwAV(Caron et al., 2021), BYOL (Grill et al., 2020), and SimSiam (Chen & He, 2020)). Moreover, these works does not have any design for learning personalized models.
sections.heading: 7 CONCLUSION
sections.text: We propose Self-supervised Federated Learning (SSFL) framework and a series of algorithms under this framework towards addressing two challenges: data heterogeneity and label deficiency. SSFL can work for both global model training and personalized model training. We conduct experiments on a synthetic non-I.I.D. dataset based on CIFAR-10 and the intrinsically non-I.I.D. GLD-23K dataset. Our experimental results demonstrate that SSFL can work reliably and achieves reasonable evaluation accuracy that is suitable for use in various applications.