text:
\section{Introduction}\label{sec:introduction}}

\IEEEPARstart{A}{lthough} Deep Learning (DL) approaches have achieved significant accomplishments in a wide range of applications \cite{suvorov2022resolution, saharia2022photorealistic, schulman2022chatgpt}, there exists several fundamental challenges that yet need to be addressed in order to enable its complete power \cite{dong2021survey}. One major problem is the difficulty in the centralization of the local data from multiple clients to one central server, leading to data sovereignty and data privacy concerns. In order to guarantee the privacy-preserving in training schemes, Federated Learning (FL) framework \cite{mcmahan2017communication} is proposed to allow multiple entities (e.g. mobile devices or large organizations) collaboratively train a DL model under the coordination of a central server. Specifically, each client's data is trained and stored locally, and only updated gradients are transferred to the server for aggregation purposes. By keeping the training data decentralized, FL approaches can alleviate many of the systemic privacy risks of traditional centralized DL methods.

However, under FL's naive implementation, since properties of the training data and local models' gradients are often stored on centralized models, there exists the possibility of user data being leaked through ones. In other words, although FL enables local training without the data exchange between the central party and local clients, private information can be divulged from training gradients through inversion attacks \cite{geiping2020inverting,wang2019beyond,yin2021see,zhao2020idlg,zhu2019deep}, leading to the high risk of users' training data being reconstructed. Additionally, the repercussions regarding the leakage of user data would be even more severe if the central party holds a full control on observing and modifying all clients' gradients. \cite{boenisch2021curious} \textcolor{red}{[HP's comment: Here you can say Boenisch \textit{et al.} [6]]} proposed that in the active-and-dishonest threat model, the adversaries can set up trap weights for recovering the clients' data with zero reconstruction loss.

With the aim at preventing the leakage of users' information, \cite{dwork2014algorithmic} \textcolor{red}{[HP's comment: Say the authors of [7] or X et al. [7]]} proposed the framework of differential privacy (DP), which is the state-of-the-art privacy-guarantee technique that constrains the exposure of DL models to private information in training datasets by adding artificial noises to the computational steps. By combining FL and DP, there are two approaches to protect the privacy of users' information: global DP (GDP) and local DP (LDP). In terms of GDP, noises are added by the central party, while in the LDP methods, each client perturbs its information locally before sending it to the central party. However, in cases of the central party is dishonest, the GDP approach cannot ensure privacy preservation since the clients would share their gradients with it. Therefore, in this research, we concentrate on the LDP scenario to enhance the protection of both the clients and the central party against sensitive information leakage. \textcolor{red}{[HP's comment: Please use more citations to support your statements]}.

Another issue in FL is the non-independent and identically distributed (non-IID), caused by the highly skewed data distribution among clients \textcolor{red}{[citation?]}. In particular, in non-IID scenarios, different models in clients are trained by different data distributions, which leads to the deterioration in the performance of the aggregated model. In the scope of our study, we focus on the cross-silo FL in which clients are large entities (e.g., companies, institutions, and hospitals), and each client possess a relatively huge amount of local data, with the main assumption that neither the computational power nor the data quantity of the clients is constrained. Considering real-world cross-silo FL settings where the data can be concurrently private and non-IID, (e.g., multiple hospitals in different locations share their patients' health records and medical scans to combat a new disease) we propose a framework that takes the problem of data non-IID and privacy concerns into account.

In this work, we propose a Personalized Privacy-Preserving Federated Learning (PPPFL) framework to overcome these challenges, allowing participants to collaborative learn from a highly-private, low-utility synthetic version of their data without worrying about the sensitive information being reconstructed or the predictive performance being affected. We also take those problems into consideration from the users' viewpoint by allowing the participating users to customize their preferred privacy level depending on their specific products and business goals and studying their incentives to participate in the protocol.
\leavevmode
\\

\textbf{Technical Contributions.} In this paper, we propose a personalized privacy-preserving framework \textcolor{red}{[HP's comment: Repeated, why did not use PPPFL as mentined above?]} for cross-silo federated learning that not only alleviate the leakage of user data but also handle the non-IID problems. To be more precise \textcolor{red}{[HP's comment: You used too much this word, please consider to use other]}, we make contributions as follows:

\begin{itemize}
    \item \st{Our proposed} \textcolor{red}{The proposed} framework provides differential privacy at the instance level, not only the user-level as in the prior works in privacy-preserving FL, which provides better privacy guarantee, especially in the active-and-dishonest threat model and cross-silo FL, while still maintaining high predictive performance at inference time. \textcolor{red}{[HP's comment: Too long and bad wording, you did used 4 commas in this sentence. Please consider to break it in two sentences.]}

    \item We leverage the off-the-shelf FedMeta \cite{chen2018federated} in terms of personalization and convergence by incorporating the users' privacy budgets and the gradient stabilization techniques into the outer optimization loop. \textcolor{red}{[HP's comment: I do not think this is a contribution. Use something and contribute somthing are very different.]}

    \item We theoretically prove the privacy and convergence guarantee of our proposed framework.

    \item \st{We provide extensive experiments on different datasets to empirically prove the superiority of PPPFL to multiple FL baselines.} \textcolor{red}{We conduct thorough experiments and in-depth ablation studies to demonstrate the \textbf{}effectiveness of the proposed approach on different datasets. Experimental results show that the proposed PPPFL consistently outperforms previous state-of-the-art FL methods}

\end{itemize}
\leavevmode
\\

The rest of this paper is organized as follows. Related works on privacy-preserving FL, Meta-learning-based methods, and non-IID challenges in FL are reviewed in Section \ref{sec:related}. In section \ref{sec:preliminary}, we provide certain background knowledge on threat models, differential privacy, data generative models, and optimization-based meta-learning methods. Section \ref{sec:method} first presents our problem formulation and details of the proposed method, then \st{perform} \textcolor{red}{performs} a rigorous analysis of its privacy guarantee and convergence. Section \ref{sec:evaluation} provides extensive experiments on multiple benchmark datasets. Finally, section \ref{sec:discussion} discusses the experimental results, some key findings, and limitations of this work and concludes the paper.\\
\\
\textcolor{red}{Dear Tuan Tran,\\
\\
I think the Introduction did not meet key requirements for a good-written and high-quality paper. First, please highlight major limitations in the Literature and show that this is no existing work solving this problem. Second, consider to highlight that your contribution is unique and this is the first time it is introduced.}

\section{Related Works}
\label{sec:related}
\noindent
\textbf{Privacy-preserving Federated Learning.}
In this work, due to the consideration of the Active-and-dishonest threat model, we focus on Local Differential Privacy (LDP) approaches, where one commonly bound the sensitivity of the participants' parameter updates. Specifically, \cite{sun2020ldp} proposed a novel method to overcome challenges in applying local differential privacy by using the adaptive range of clients' weights to achieve better-aggregated model performance, especially in deeper models, and leveraging parameter shuffling as an extra layer of privacy protection. Meanwhile, \cite{truex2020ldp} allowed the individual clients to define local privacy guarantees by developing an FL training approach and a parameter sharing method by applying Condensed Local Differential Privacy with Exponential Mechanism \cite{gursoy2019secure} to obtain LDP guarantees with some degrees of interpretation. In 2021, user-level differential privacy was proposed by \cite{wei2021user}, in which DP-SGD with Moment Accountants \cite{abadi2016deep} is leveraged. Moreover, they also theoretically demonstrated the significance of Communication rounds in boosting the performance of the learning model. As a result, a communication rounds discounting method is designed to minimize the tradeoff between the privacy budget and the convergence performance.

Regarding cross-silo settings, \cite{mahawaga2022local} leveraged RAPPOR \cite{erlingsson2014rappor} algorithm and optimized unary encoding approach on clients' data representations, allowing federated-learning without the need for a trusted server. This proposed method can ensure a high bit preservation during binary string randomization by randomizing one-half of the binary string. Recently, \cite{hu2022federated} designed a differentially-private FL model with the idea taken from sparsified model perturbation, improving not only the privacy-accuracy trade-off but also communication efficiency with a large number of participants.
\leavevmode
\\

\textbf{Solving the non-IID problem in Federated Learning.}
Due to the heterogeneity in participants' data distributions, having a single global model that can fit all the models among the parties is difficult \cite{kairouz2021advances}. When it comes to the non-identical data problem, especially in cross-silo FL, where the data quantities and computational power are available, one commonly chosen strategy is to use different models for different participants, which means that each participant will have its data distribution. Early work is conducted by \cite{smith2017federated}, who cast the learning procedure of multiple separate local models as a multi-task learning problem to deal with challenges related to statistics and increase the communication capacity of each participant. With another approach,
\cite{t2020personalized} utilized the convexity-preserving and smoothness of Moreau envelope \cite{moreau1963proprietes} as a regularized loss function. This method not only helps to simplify the optimization of global model and local, personalized models but also provide rigorous and quick convergence.

To handle the non-IID problem in personalized cross-silo FL, \cite{huang2021personalized} proposed a mechanism called attentive message passing, which permits each client to pass its personalized model as a message to its corresponding cloud model maintained on the cloud server. Moreover, a heuristic method based on Euclidean distance is also proposed to improve the performance of the method. Meanwhile, in \cite{shamsian2021personalized}, a Hypernetwork \cite{ha2016hypernetworks} is utilized to gain unique embedding vectors sent by clients and then produce corresponding personalized weights, helping ensure the uniqueness of each client's model and reducing the communication cost. In the scope of our research, our inspiration stems from these works \cite{fallah2020personalized, chen2018federated, jiang2019improving} with the emphasis placed on the connection between FL and Meta-learning \cite{younger2001meta}, especially with MAML \cite{finn2017model}.

\section{Preliminaries}
\label{sec:preliminary}
In this section, we provide the details of threat models in privacy when it comes to the existence of an active-and-dishonest server. Next, we provide some background knowledge on Differential Privacy (DP) and the Local DP guarantees in Federated Learning. Finally, we briefly discuss Optimization-based Meta-learning through the MAML \cite{finn2017model} framework.

We also summarize the notations used in this paper in Table \ref{tab:notations}.

\begin{table}[]
\caption{Main notations used in the paper.}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{l|l}
\hline
$\theta$                        &   Global model's parameters               \\ \hline
$\theta_i$                      &   Local model's parameters in client $i$  \\ \hline
$D_i$                           &   Secret (private) dataset in client $i$            \\ \hline
$\mathcal{G}_i$                      &   Local DP data generator of client $i$ trained from $D$ \\ \hline
$\mathfrak{D}_i$                &   Synthetic dataset of client $i$ generated by $\mathcal{G}_i$\\ \hline
$\epsilon_i$                    &   Privacy budget chosen by client $i$ when training $\mathcal{G}_i$       \\ \hline
$\alpha^{*}$                    &   Inner learning rate when training local model $\theta$ using synthetic data $\mathfrak{D}$   \\ \hline
$\beta^{*}$                     &   Outer learning rate when server aggregate local models $\theta$ \\ \hline

\end{tabular}
}
\label{tab:notations}
\end{table}

\subsection{Threat models}
\textbf{Semi-honest server.} In this type of server, all participated parties follow the protocol to perform the computation task but they may try to infer the data of other parties from the received intermediate computation results \cite{li2021trustworthy}. It has been shown by \cite{geiping2020inverting}, in the data recovery attacks \cite{7958568}, the gradients of any fully connected (FC) layer followed by a ReLU activation function $\mathbf{y} = \textsc{ReLU}(W\mathbf{x} + \mathbf{b})$ contain the training data point $\mathbf{x}$. Formally, let $\mathbf{w}_i^T$ be the corresponding row in the weight matrix and $b_i$ be the corresponding component in the bias vector, the data point $\mathbf{x}$ can be reconstructed by computing $\mathbf{x}^{T}=\left(\frac{\partial \mathcal{L}}{\partial b_{i}}\right)^{-1} \frac{\partial \mathcal{L}}{\partial \mathbf{w}_{i}^{T}}$, as long as the neuron $i$ of that FC layer is fired (i.e., $\mathbf{w}_i^T \mathbf{x} + b_i > 0$). Consequently, a passive attacker can extract roughly 20\% of arbitrary data points from a batch size $B =  100$ for 1000 neurons on ImageNet.

\textbf{Active-and-dishonest server.}
Based on the data leakage through gradients of the FC layer, \cite{boenisch2021curious} has raised the privacy concerns of FL to a greatly alarming level when the central server can not only observe the gradients but also modify the shared weights as a trap before sending to the users. Specifically, if the weight row $\mathbf{w}_i$ is adversarially initialized so that every neuron in the FC layer activates only one input data point per mini-batch, the adversary can perfectly reconstruct the input data points in the mini-batch. Furthermore, this gradient leakage can be generalized for convolutional neural networks (CNNs) by combining the identity convolution kernels as the trap weights and using reasonable padding and stride to effectively transmit the input image down into the FC layer.

\subsection{Differential Privacy}

\begin{definition}
    \label{def:dp}
    $(\epsilon-\delta)$ Differential Privacy

    Let a randomized algorithm $M$ be given access to a dataset $D$, and let $D'$ be a neighboring dataset that differs from $D$ in at most one element. Then, $M$ satisfies $(\epsilon, \delta)$-differential privacy if for all outputs $O$ of $M$ and for all sets $S$ of possible outputs, the following inequality holds

    \begin{equation}
    \begin{array}{rl}
        \label{eq:epsilon_delta_dp}
        Pr[M(D) \in S] &\leq \exp(\epsilon) * Pr[M(D') \in S] + \delta.
    \end{array}
    \end{equation}

    This definition captures the idea that the probability of any particular algorithm output should not be significantly affected by the presence or absence of any individual's data in the dataset. The parameters $\epsilon$ and $\delta$ control the strength of the privacy guarantee, with smaller values of $\epsilon$ and $\delta$ indicating stronger privacy guarantees.

\end{definition}

\begin{definition}
    \label{def:rdp}
    $(\lambda, \alpha)$-Renyi Differential Privacy

    A randomized mechanism $\mathcal{M}$ guarantees $(\lambda, \alpha)$-RDP with $\lambda>1$ if, for any neighboring datasets $D$ and $D'$, the following inequality holds

    \resizebox{0.9\columnwidth}{!}{%
    $$D_\lambda\left(\mathcal{M}(D) | \mathcal{M}\left(D^{\prime}\right)\right)=\frac{1}{\lambda-1} \log \mathbb{E}*{x \sim \mathcal{M}(D)}\left[\left(\frac{\operatorname{Pr}[\mathcal{M}(D)=x]}{\operatorname{Pr}\left[\mathcal{M}\left(D^{\prime}\right)=x\right]}\right)^{\lambda-1}\right] \leq \alpha$$
    }
    \end{definition}

\begin{theorem} (From RDP to DP)

    If a mechanism $\mathcal{M}$ guarantees $(\lambda, \alpha)$-RDP, then it also guarantees $\left(\alpha+\frac{\log 1 / \delta}{\lambda-1}, \delta\right)$-differential privacy for any $\delta \in(0,1)$.

\end{theorem}

\noindent
\textbf{Local DP and Global DP.}
\textcolor{blue}{
In \st{Federated Learning} FL, global differential privacy (global DP) refers to a privacy-preserving technique in which a trusted aggregator adds noise to multiple users' data or parameter updates to protect their privacy. The noise is added using a DP mechanism, which is designed to ensure that the noise is sufficient to prevent the identification of any individual user's data, even when the aggregator has access to the data of all users.}

However, when the central server, or aggregator, is active and dishonest, it \textcolor{blue}{\textcolor{red}{\st{may be possible for the server to omit the additional noise meant to protect the users' privacy. This}}  could \textcolor{red}{\st{potentially allow the server to}} identify the data of individual users. \textcolor{red}{\st{compromising their privacy. In such cases, it may not be safe to send the data to a central server, as the server may not be trusted or have sufficient safeguards to protect the privacy of the individuals whose data is being collected. To protect the privacy of the individuals whose data is being collected}} Hence, local DP adds noise to the data locally before it is sent to the central server \textcolor{red}{\st{. This ensures that the privacy of the individuals is protected, even if the central server is untrusted or if the data is}} or transmitted over an insecure network.}

\textbf{User-level DP and instance-level DP in FL.}
User-level DP \st{on one hand,} is a variant of local DP \textcolor{red}{\st{that is designed to protect the privacy of individual users rather than individual data samples. It is typically}} used when the data consists of a smaller number of samples associated with specific users, and the focus is on protecting the privacy of the users rather than the privacy of the individual data points.
\textcolor{blue}{
On the other hand, instance-level differential privacy is generally more robust than user-level DP because it provides stronger privacy guarantees for individual data samples. \textcolor{red}{\st{This is because instance-level DP is designed to protect the privacy of individual data points rather than individual users, and it provides stronger guarantees against the risk of an adversary learning sensitive information about specific data points from the released dataset.}} Particularly, instance-level DP ensures that the probability of an adversary learning sensitive information about a specific data point is bounded by a small privacy budget. \textcolor{red}{\st{This is typically stronger than the privacy guarantees provided by user-level DP, which bounds the probability of an adversary learning sensitive information about a specific user rather than a specific data point.} } Moreover, instance-level DP is \st{typically} easier to implement and apply in practice than user-level DP, as it does not require tracking or identifying specific users or user groups in the dataset, thus making it more flexible and easier to use, especially in the context of cross-silo FL. }

\subsection{Differentially Private Data Generation}

Deep generative models \cite{goodfellow2020generative, Kingma2014AutoEncodingVB, ho2020denoising} are now the preferred choice for multiple purposes, ranging from AI for creativity \cite{rombach2022high, yang2022pastiche}, handling imbalanced data \cite{ren2019ewgan}, or enhancing representation learning \cite{chen2016infogan}. Due to the high expressiveness of deep neural networks in modeling the data distribution, DL-based generative models can output high-fidelity samples. However, those synthetic samples contain sensitive information \cite{chen2020gan, lin2021privacy} from the training data. This leads to an active field of research where the generative models are combined with \st{Differential Privacy} DP in immediate computational steps. By doing so, the DP-generators can synthesize the protected samples against membership inference attacks \cite{7958568} and gradient inversion attacks \cite{geiping2020inverting}.

\textcolor{blue}{\textcolor{red}{\st{Differential Privacy (DP) can be used to protect the privacy of individuals whose data is used to train}} Generative Adversarial Networks (GANs) \st{GANs} are \st{a type of} machine learning models that consist of a generator and a discriminator, which are trained to play a minimax game to generate synthetic data that is similar to a given dataset.}
One way to incorporate DP into the training of GANs is to use DP-Stochastic Gradient Descent (SGD) \cite{abadi2016deep}, which adds calibrated Gaussian noise to the gradients of the discriminator \cite{torkzadehmahani2019dp, xie2018differentially}. Another option is to use the Privacy-Aware Teacher and Ensemble (PATE) framework \cite{papernot2018scalable}, which involves aggregating the opinions of multiple teacher discriminators in order to train a student discriminator \cite{jordon2018pate} or student generator \cite{long2019scalable} with better privacy budget usage.

One of the key benefits of using DP with GANs is the post-processing property \cite{dwork2014algorithmic} of DP mechanisms, which states that the composition of an $(\epsilon-\delta)-DP$ mechanism is also $(\epsilon-\delta)-DP$. This property not only applies to the generator, but also to the generated data, as shown in \ref{sec:privacy_analysis}. This means that the privacy of the individuals whose data is used to train the generator is preserved, even when the generated data is used for downstream tasks.

\subsection{Optimization-based Meta-learning}

Optimization-based meta-learning methods \cite{finn2017model} aims for meta-training an algorithm $\mathcal{A}$ parameterized by $\theta$, which can perform well on a new task $T+1$ through learning from existing $T$ tasks with the assumption that all the tasks share the same structure.
In supervised learning settings, a task $t$ contains i.i.d. observations $X_t$ and their corresponding targets $y_t$. To evaluate a meta-learning algorithm, one usually split each task's data $(X_t, y_t)$ into the training (support) set $(X_t^{tr}, y_t^{tr})$ and validation (query) set $(X_t^{val}, y_t^{val})$.

\textbf{Inner loop} mapping from prior $\theta_0$ to task-specific predictor $\theta_t$

\begin{equation}
    \label{eq:inner_loop}
    \theta_t = \mathcal{A}(\theta_0; X_t, y_t)
\end{equation}

\textbf{Outer loop} solving empirical-risk minimization problem over many tasks to find the best prior

\begin{equation}
    \label{eq:outer_loop}
    L(\theta_0) = \frac{1}{T}\sum^{T}_{t=1}{\mathcal{L}(\mathcal{A}(\theta_0; X_t^{tr}, y_t^{tr}; X_t^{val}, y_t^{val}))}
\end{equation}

\textbf{Meta-training} is to learn a model that can adapt to new tasks quickly and efficiently using only a small number of gradient descent steps without the need for extensive training data. Formally, in the context of optimization-based meta-learning, the training objective function can be formulated as follows:

\begin{equation}
    \label{eq:meta_training}
    \resizebox{0.9\columnwidth}{!}{%
    $$L^{train}(\theta_0) = \frac{1}{T}\sum_{t=1}^{T}{\mathcal{L} \left( \mathcal{A}\left(\theta_0; X_t^{tr}, y_t^{tr} \right); X_t^{val}, y_t^{val} \right)} $$
    }
\end{equation}

\textbf{Meta-testing} is the process of evaluating the performance of a model that has been meta-trained  on a set of new tasks. During meta-testing, the model is presented with a novel task that is similar but has not been seen during meta-training. The empirical risk on novel task $T+1$ can be formulated as follows

\begin{equation}
    \label{eq:meta_test}
    \resizebox{0.9\columnwidth}{!}{%
     L^{test}(\theta_0) \\
     = \mathbb{E}_{(X_{T+1}, y_{T+1}, (x', y') ~ p_{T+1})}\left[ \mathcal{L}(\mathcal{A}(\theta_0; X_{T+1}, y_{T+1}); x', y') \right]
     }
\end{equation}

\section{Methodology}
\label{sec:method}

In this section, we present a formal problem formulation in which the goal is to enhance privacy in personalized Federated Learning (FL). We then proceed to describe our proposed solution to this problem, the PPPFL framework, in detail. This includes a step-by-step description of the process, beginning with the training of local models on the client side and culminating in the federated collaboration between the clients and the server. Finally, we provide theoretical proof demonstrating the privacy and convergence guarantees of the PPPFL framework.

\subsection{Problem Formulation}
\noindent
\textbf{Privacy-preserving FL} Given a set of $K$ clients $\{ c_1, \dots, c_K \}$ sampled from a client distribution $\mathcal{C}$ and an un-trusted server $S$. The $i^{th}$ client has access to a private local dataset $D_i$; these private data sets are non-IID. Particularly, we study the common scenario where the local data distribution $\Pr(x_i, y_i) = \Pr(x_i \mid y_i) \Pr(y_i)$ with the label distributions $\Pr(y_i)$ is different among clients. These local private datasets $D_i$ are divided into training and test subset for evaluation purpose $D_i^{tr}$ and  $D_i^{val}$, respectively, with $D_i = D_i^{tr} \cup D_i^{val}$. Due to the concern of the active-and-dishonest server $S$, the private datasets will be kept locally and never be used for collaborative training. Instead of directly using the private data for the FL protocol, the clients train their own personalized private data generator $\mathcal{G}_i$, and then use the synthetic datasets $\mathfrak{D}_i^{tr}$ and $\mathfrak{D}_i^{val}$ with $\mathfrak{D}_i = \mathcal{G}_i(D_i)$ for collaboration. The privacy budget $\epsilon_i$ in each data generator $\mathcal{G}_i$ is customized by the clients.

\noindent
\textbf{Personalized FL} We aim to provide the clients the personalized models to match their data and their preferred privacy level by finding an initial shared model that the users can easily adapt to their local dataset later. Specifically, on the server side, the goal is using synthetic datasets $\mathfrak{D}_i$ to meta-learn a well-initialized model parameterized by $\theta$ from the updates sent by the clients, so as the trained model can be later adapted to the client's private data $D_i$. The support set $\mathfrak{D}_i^{tr}$ and query set $\mathfrak{D}_i^{test}$, and the meta-model will be locally adapted to the private data set $D_i^{tr}$ and evaluated on $D_i^{test}$ by the clients.

Considering the meta-train tasks and meta-test tasks $t \sim \mathcal{T}$ are now the clients $i \sim \mathcal{C}$ with the synthetic $\mathfrak{D}_i$ and private data $D_i$, respectively. The meta-training objective function in Eq. \ref{eq:outer_loop} can be re-written as:

\begin{equation}
L(\theta_0) = \mathbb{E}_{c \sim \mathcal{C}}\left[\mathcal{L}\left(\mathcal{A}\left({\theta_0}; \mathfrak{D}_{c}^{tr}\right); \mathfrak{D}_i^{val} \right)\right]
\end{equation}

\subsection{Proposed Framework}
We first introduce the data flow in each client and the central server. Then, we present our techniques to overcome the challenges of gradients instability \cite{antoniou2018train} when using MAML \cite{finn2017model} as the optimization-based meta-learning algorithm, and the computational steps of the chosen DP-GAN architecture \textsc{DataLens} \cite{wang2021datalens}.

\subsubsection{Method Overview}
The goal of the PPPFL framework is to leverage differential private data generated from DP-GANs in terms of utility by collaboratively training a meta-global model that can be later adapted to the client's secret data. The flow of our proposed framework includes 3 main phases which are illustrated in Fig. \ref{fig:framework}.
In the first stage, to locally synthesize private data given the secret dataset, each participant will train their own DP-synthetic data generation model using Algorithm \ref{alg:datalens}. The DP-guarantee data will be alternatively used to collaborate with each other. Due to privacy concerns, the secret data is kept locally for evaluation purposes only when the collaboration stops.
Next, during the Federated training phase (as shown in Algorithm \ref{alg:federatedTraining}), at the beginning of each communication round between clients and the server, each client will download the weights of a global model.
Then for every communication round, by training local models on synthetic data, the clients perform gradients decent updates on the synthetic data generated by their own DP-GANs, specifically the training sets, with respect to the global model's parameters. On one hand, the clients will then get feedback from the query set by calculating gradients of their local models with respect to their held-out validation sets (as shown in Algorithm \ref{alg:clients}) and upload these signals to the server. On the other hand, the server performs updates to the global model using the gradients of clients' local models and the received feedback signals (as shown in Algorithm \ref{alg:server}).
After the collaboration rounds stop, the clients get emerged from the server and perform local adaptation from their latest model downloaded from the server with respect to their local secret datasets (as shown in Algorithm \ref{alg:clients}) to get better inference performance.

\begin{figure}
    \centering
    \includegraphics[width=0.95\linewidth]{figures/PPPFL_new.png}

    \caption{The overall architecture of the PPPFL framework consisting three consecutive stages. First, each participant locally synthesizes DP-guaranteed dataset $\mathfrak{D}$ by training their own DP-data-generator $\mathcal{G}$ from secret dataset $D$. Next, in the federated training process, the synthetic data, which have been respectively separated into a training set $\mathfrak{D}^{tr}$ and held-out validation set $\mathfrak{D}^{val}$, will be used to update the global model parameters $\theta$. Lastly, the clients get emerged from the federated training rounds and perform updates to their local model parameters $\theta^{*}$ based on their secret training data $D^{tr}$ and evaluate their newly locally adapted model with respect to the secret validation set $D^{val}.$}
    \label{fig:framework}
\end{figure}

\subsubsection{Client-side computations}

\begin{algorithm}[H]
    \caption{Computational steps on the clients}
    \label{alg:clients}
    \begin{algorithmic}[1]
    \renewcommand{\algorithmicrequire}{\textbf{Input:}}
    \renewcommand{\algorithmicensure}{\textbf{Output:}}
    \REQUIRE client data $D$, client model $\theta$, fine-tuning steps $\gamma$
    \ENSURE  adapted model $\hat{\theta}$
    \\ \textit{// Before collaborative training}
    \FOR {each client $c \in \mathrm{C}$}
        \STATE{$\mathcal{G}_{c}$ \gets \textsc{DP-GAN Training}($D_{c}$)} \\
        \tcc{Training the DP data generator}
        \STATE{(i) $D_{c}$ \gets $D^{tr}_{c} \cup D^{val}_{c}$} \\
        \tcc{Locally data partitioning}
        \STATE{(ii) $\mathfrak{D}^{tr}_{c} \gets \mathcal{G}_{c}(D^{tr}_{c}),   \mathfrak{D}^{val}_{c} \gets \mathcal{G}_{c}(D^{val}_{c})$ } \\
        \tcc{Synthesizing DP data}
    \ENDFOR
    \STATE{\textsc{Federated Training}}

    \\
    \textit{// After collaborative training}
    \FOR{each client $c \in Q$ }
        \STATE{(iii) $\hat{\theta}_{c} \gets \theta_{c} - \gamma \ast\nabla_{\theta_{c}}\mathcal{L}_{c}$

        $\mathcal{L}_{D^{tr}_{c}}(\theta_{c}) \gets \frac{1}{\mid D^{tr}_{c} \mid}\sum_{\left(x,y\right)\in D^{tr}_{c}}{\ell(f_{\theta_{c}}\left(x\right),\ y)}$} \\
        \tcc{Locally fine-tuning}
\ENDFOR
\end{algorithmic}
\end{algorithm}

In this framework, besides the common FL gradients sharing with the server, the clients need to compute two extra steps, which are described in Algorithm \ref{alg:clients}. First, it is necessary to use instance-level DP to fully protect privacy in the presence of an untrusted server by training a DP data generator before collaboration. Second, the clients will perform local adaptation after receiving the well-initialized parameters sent by the server to their private data. We assume these additional steps are trivial, considering the clients are organizations with high computational power.

\subsubsection{Server-side computations}
Instead of gradually aggregating the parameters of the clients, as usually seen in the common FL settings \cite{mcmahan2017communication}, the central server in PPPFL considers the clients' synthetic data as supervised signals for the meta-train task (as described in Eq. \ref{eq:meta_training}, which aims for the achieved model can be easily adapted for the meta-test task on clients' sensitive data later. Regarding the personalized variants DP-GAN with different privacy budgets $\epsilon$ chosen by the users, we leverage the FedMeta \cite{chen2018federated} algorithm in terms of having the clients' privacy budget incorporated. The objective function in PPPFL while having multiple $\epsilon$ can be re-written as:

\begin{equation}
    \label{eq:pppfl}
    \resizebox{0.9\columnwidth}{!}{%
    L(\theta_0) = \mathbb{E}_{c \sim \mathcal{C}}\left[\frac{e^{\frac{\epsilon_{c}}{\rho}}}{\sum_{j=1}^{K} \frac{e^{\epsilon_{j}}}{\rho}} \mathcal{L}\left(\mathcal{A}\left({\theta_0}; \mathfrak{D}_{c}^{tr}\right); \mathfrak{D}_i^{val} \right)\right]
    }
\end{equation}
with $$\rho$$ is a hyperparameter that denotes the temperature of the softmax smoothing function over the clients' budgets.

\subsubsection{DP-GAN Training}
In this paper, we use the state-of-the-art DP data generator \textsc{DataLens} where the consensus of multiple teacher discriminators $\mathcal{D}_1, \mathcal{D}_2, \ldots, \mathcal{D}_N$ guide the data generator $\mathcal{G}$ follow the \textsc{PATE} framework \cite{papernot2018scalable}. The curated Gaussian noise $\mathcal{N}(0, \sigma)$ is added into the computations during the teacher aggregation step, which is described in Algorithm \ref{alg:datalens} and \ref{sec:privacy_analysis}.

\begin{algorithm}[H]
    \caption{Aggregations on the server}
    \label{alg:server}
    \begin{algorithmic}[1]
    \renewcommand{\algorithmicrequire}{\textbf{Input:}}
    \renewcommand{\algorithmicensure}{\textbf{Output:}}
    \REQUIRE initial model $\theta_0$, outer learning rate $\beta^*$
    \ENSURE  aggregated model $\theta$
    \STATE{Initialize global model $\theta \gets \theta_0$}
    \FOR {each communication round $t=1,2,\dots$}
        \STATE{Sample a set $U_t$ of $m$ clients and distribute $\theta_0$ for the sampled clients}
        \FOR {each client $c \in U_t$ in parallel}
            \STATE{$g_i$ \gets \textsc{Federated Training($\theta_i$)}} \\
            \STATE{$\theta \gets \theta - \frac{\beta^*}{m}\sum_{c\inU_t}g_i$} \tcc{Update the global model}
        \ENDFOR
    \ENDFOR
\end{algorithmic}
\end{algorithm}

\SetKwProg{Fn}{Function}{}{}

\begin{algorithm}[H]
    \caption{Federated Training}
    \label{alg:federatedTraining}
    \DontPrintSemicolon
    \begin{algorithmic}[1]
    \renewcommand{\algorithmicrequire}{\textbf{Input:}}
    \renewcommand{\algorithmicensure}{\textbf{Output:}}
    \REQUIRE client synthetic data $\mathfrak{D}_i^{tr}, \mathfrak{D}_i^{val}$, initial model $\theta_0$, inner learning rate $\alpha^*$
    \ENSURE  updated model $\theta^*$
    \STATE{Initialize the local model $\theta \gets \theta_0$}
    \FOR {each communication round $t=1,2,\dots$}
        \STATE{\textit{ClientUpdate($\theta_i$, $\mathfrak{D}_i^{tr}$)}}
        \STATE{$\mathcal{L}_{\mathfrak{D}_{c}}^{val}\left(\theta_{c}\right)\gets\frac{1}{\left|\mathfrak{D}_{c}^{val}\right|}\sum_{ \left( x^\prime,y^\prime \right) \in \mathfrak{D}_{c}^{val}} \hairsp\ell\left(f_{\theta_i}\left(x^\prime\right),y^\prime\right)$}  \\
        \tcc{Compute the query loss}
        \STATE{$g_i  \gets \nabla_\theta_{i} \mathcal{L}_{\mathfrak{D}_i^{val}} \left(\theta_i\right)$ }
        \STATE{Uploading $g_i$ to the server}
    \ENDFOR \\
    \Fn{ClientUpdate($\theta_i$, $\mathfrak{D}_i^{tr}$)}{
        $\mathcal{L}_{\mathfrak{D}_{c}}^{tr}\left(\theta_{c}\right)\gets\frac{1}{\left|\mathfrak{D}_{c}^{tr}\right|}\sum_{ \left( x,y \right) \in \mathfrak{D}_{c}^{tr}}\hairsp\ell\left(f_{\theta_i}\left(x\right),y\right)$ \\
        $\theta_i  \gets \theta_i - \alpha^* * \nabla_\theta_i \mathcal{L}_{\mathfrak{D}_i^{tr}}$ \\

            }

\end{algorithmic}
\end{algorithm}

\begin{algorithm}[H]
    \caption{DP-GAN Training}
    \label{alg:datalens}
    \begin{algorithmic}[1]
    \renewcommand{\algorithmicrequire}{\textbf{Input:}}
    \renewcommand{\algorithmicensure}{\textbf{Output:}}
    \REQUIRE client private dataset $D_i$, $N$ discriminators $\mathcal{D}_1, \ldots, \mathcal{D}_N$
    \ENSURE DP generator $\mathcal{G}$
    \STATE Partitioning the sensitive dataset $D_i$ into non-overlapping subsets $\mathrm{d}_{1}, \mathrm{~d}_{2}, \ldots, \mathrm{d}_{\mathrm{N}}$ an associate a teacher discriminator $\mathcal{G}_{i}$ to each set.
    \STATE Training the teachers: using the discrimination loss $\mathcal{L}_{\mathcal{G}}=-\log \mathcal{G}(x)-\log (1 -  \mathcal{G}\left(\mathcal{D}^{\prime}(z)\right)$ with $\mathcal{D}$ denotes the student generator $\mathrm{x}$ is sampled from the subsets $\mathrm{d}_{\mathrm{i}}, z$ is the noise sample.
    \STATE Gradients Compressing: the teacher's dense gradients vectors $g^{(i)}$ into sparsified $\{-1,0,1\}$ valued vectors $\tilde{g}^{(i)}$.
    \STATE DP Aggregating: the teacher's gradients $G=$ $\left\{\tilde{g}^{(1)}, \ldots, \tilde{g}^{(\mathrm{N})}\right\}$ with Gaussian noise $\mathcal{N}\left(0, \sigma^{2}\right)$ and use them to train the student generators with the generator loss $\tilde{\mathcal{L}}_{\mathcal{G}}=$ $\frac{1}{m} \sum_{j=1}^{m}\left(\mathcal{D}\left(z_{j}\right)-\hat{x}_{j}\right)^{2}$, where $\hat{x}_{j}=\mathcal{D}\left(\mathrm{z}_{j}\right)+\gamma \tilde{g}$ is the synthetic data plus the aggregated $D$

\end{algorithmic}
\end{algorithm}

\subsubsection{Stabilizing MAML algorithm}
As previously noted by Antoniou et al. \cite{antoniou2018train}, the optimization process in Model-Agnostic Meta-Learning (MAML) can be unstable due to the multiple times that differentiation is performed through the model. This instability is illustrated in Fig. \ref{fig:maml_before}, which shows that the original MAML model diverges after just a few communication rounds. To address this issue, we propose a modified version of MAML that employs Model Exponential Moving Average in the inner loop, followed by Cosine Annealing Learning rate scheduling in the outer loop. This approach is inspired by previous work on improving generalization in DL over Stochastic Gradient Descent (SGD) in general \cite{izmailov2018averaging, tarvainen2017mean, huang2017snapshot} and improving stability in MAML in specific \cite{antoniou2018train}. Formally, the model parameters $\theta_t$ of the clients at communication round $t$ are set to be an Exponential Moving Average (EMA) of the previous round's model parameters: $\theta_t := \zeta \theta_t + (1-\zeta) \theta_{t-1}$, where $\zeta$ is the momentum hyperparameter. The learning rate $\beta^$ for the outer optimization loop is calculated as $\beta^(t)=f(\bmod (t-1,\lceil T / M\rceil)) $, where $t$ is the current round of the total $T$ communication rounds, $M$ is the number of cosine annealing cycles, and $f$ is a monotonically decreasing function. By combining these techniques, the global model is able to traverse multiple local optima while maintaining stability during training.

\begin{figure}
  \begin{subfigure}[b]{0.8\columnwidth}
    \includegraphics[width=\linewidth]{figures/synthetic.pdf}
    \caption{The diverging learning curve.}
    \label{fig:maml_before}
  \end{subfigure}
  \hspace{0.5 cm}
  \begin{subfigure}[b]{0.8\columnwidth}
    \includegraphics[width=\linewidth]{figures/synthetic_after.pdf}
    \caption{The stabilized learning curve.}
    \label{fig:maml_after}
  \end{subfigure}

  \caption{The improvement on convergence (bottom) compared to the original MAML (top) indicated by the total cross-entropy loss of the support set and query set over communication rounds. The top figure indicates a diverging learning curve when the gradients keep accumulating while the bottom figure illustrates fast convergence after 10 communication rounds. The experiment is conducted on the CIFAR-10 dataset.}
\end{figure}

\subsubsection{Personalization}

In our approach, we consider the need for customization and personalization, recognizing that different clients may have different business requirements for data sharing. Rather than having the server control the level of user privacy, we allow clients to choose their own privacy levels based on their actual needs. This enables clients to independently and effectively select their preferred privacy budgets $\epsilon$ and incorporate them directly into the DP-GAN training procedure, while still being able to control the outcomes of the training process without having to wait for the collaborative model to converge. Additionally, this information about the client's privacy budgets is used in our proposed framework to improve the performance of the client's models.

\subsection{Privacy Analysis \tablefootnote{%
The following analysis is based on work by Wang et al. \cite{wang2021datalens} and has been reproduced with the written consent of the authors and the publisher for the purpose of providing clarity to the reader.}
}
\label{sec:privacy_analysis}
In our chosen DP-generative algorithm \cite{wang2021datalens}, briefly described in Algorithm \ref{alg:datalens}, the gradients of the student generator are obtained by aggregating the sparsified gradients of the teachers and voting. Specifically, the gradient set $G=$ $\left{g^{(1)}, \ldots, g^{(\mathrm{N})}\right}$ of $N$ is quantized using top-$k$-sparsification \cite{shi2019understanding} to convert the real-valued gradient vector into a sparse {-1, 1} vector $\tilde{g}^{(i)}$ with $k$ non-zero entries.

The Gaussian mechanism can be applied to the sum aggregation function defined as $f_{\text{sum}}(\tilde{\mathcal{G}})=\sum_{i=1}^N \tilde{\mathbf{g}}^{(i)}$ for a set of compressed teacher gradient vectors $\tilde{G}=\left(\tilde{\mathbf{g}}^{(1)}, \ldots, \tilde{\mathbf{g}}^{(N)}\right)$, where $\tilde{\mathbf{g}}^{(i)}$ is the compressed gradient of the $i$-th teacher. The resulting output is $\tilde{\mathbf{G}}\sigma f{\text{sum}}(\tilde{\mathcal{G}})=f_{\text{sum}}(\tilde{\mathcal{G}})+\mathcal{N}\left(0, \sigma^2\right)=\sum_{\tilde{\mathbf{g}} \in \tilde{\mathcal{G}}} \tilde{\mathbf{g}}+\mathcal{N}\left(0, \sigma^2\right)$. The Gaussian mechanism guarantees $(\lambda, \alpha)$-Renyi Differential Privacy (RDP) for any real-valued function $f$.

\begin{lemma} For any neighboring top- $k$ gradient vector sets $\tilde{\mathcal{G}}, \tilde{\mathcal{G}}^{\prime}$ differing by the gradient vector of one teacher, the $\ell_2$ sensitivity for $f_{\text {sum }}$ is $2 \sqrt{k}$.
\end{lemma}

According to Theorem 1, the Gaussian mechanism $\mathrm{G}\sigma f$ to a function $f$ with $\ell_2$ sensitivity results in a $(\lambda, s^2 \lambda /(2 \sigma^2))$-RDP guarantee.
In order to determine the RDP guarantee for the Gaussian mechanism applied to the sum aggregation function $\tilde{\mathrm{G}}\sigma f_{\mathrm{sum}}(\tilde{\mathcal{G}})$, it is necessary to calculate the $\ell_2$ sensitivity of this function. According to Lemma 1, the $\ell_2$ sensitivity of $f_{\text{sum}}$ for any neighboring top-$k$ gradient vector sets $\tilde{\mathcal{G}}, \tilde{\mathcal{G}}^{\prime}$ that differ by a single gradient vector is equal to $2 \sqrt{k}$. This value can be used to determine the RDP guarantee for the Gaussian mechanism applied to the sum aggregation function.

\subsection{Convergence Analysis}
\label{sec:convergence_analysis}

\section{Evaluation}
\label{sec:evaluation}
In this section, we illustrate the experimental evaluation of our proposed framework for generating differentially private data with privacy-preserving. We evaluate our method on different benchmark datasets to demonstrate its effectiveness over the baseline methods.

\subsection{Experiment Setup}
\noindent
\textbf{Datasets}. We use four benchmark datasets, including MNIST \cite{deng2012mnist}, Fashion-MNIST \cite{xiao2017fashion}, CIFAR-10 \cite{krizhevsky2010convolutional}, and CIFAR-100, to evaluate our proposed method. MNIST and Fashion-MNIST have $60000$ training examples and $10000$ testing examples, and all of them are grayscale images of $28 \times 28$ dimensions. We also utilize two color-image datasets, CIFAR-10 and CIFAR-100, to demonstrate the performance of the proposed method. Both datasets have $50000$ in training images and $10000$ in testing images.
\\

\noindent
\textbf{Baselines}. We compare our proposed method with the most popular FL algorithm FedAvg \cite{mcmahan2017communication}, and state-of-the-art FL algorithms under non-IID data, including FedNova \cite{wang2020tackling}, FedProx \cite{li2020federated}, SCAFFOLD \cite{karimireddy2020scaffold}, FedMeta \cite{chen2018federated}, and SDA-FL \cite{huang2021personalized}.

\noindent
\textbf{Federated Learning settings} We simulate the FL scenario with $5$ clients and $30$ communication rounds. In each communication round, all of the clients participate in the protocol. The inner learning rate $\alpha^*$ and outer learning rate $\beta^*$ for the meta-training procedure are respectively set to $1e-4$ and $3e-4$, respectively. The used optimizer is Adam \cite{kingma2014adam} and batch size is $64$.

\noindent
\textbf{DP data generation model}
We follow the hyperparameter choices by \cite{wang2021datalens}. Specifically, for the MNIST Handwritten Dataset, the DP GANs are trained using privacy budget $\epsilon \in { 1, \dots, 10} $. The sensitivity $\delta$ is set to $1e-5$, $top-k$ value equals to $300$. The noise parameter $\sigma = 5000$ and voting threshold $\beta = 0.7$.
\\
\noindent
\textbf{Non-IID simulation}
We simulate the common non-IID scenario where the label distribution is skewed. The label distribution $q \sim Dirichlet(\tau)$ with $\tau = 1.0$.
\\
\noindent
\textbf{Metrics}
\noindent
We follow the work by Huang et. al. \cite{huang2021personalized} on employing Best Mean Test Accuracy (BMTA) as the metric of averaging the testing accuracy on all clients and selecting the best value across the training communication rounds. We further use Best Mean Test F1-score (BMT-F1) as an additional metric for better evaluation when it comes to the imbalance made by the non-IIDness of the data.
Formally, given the accuracy score and F1-score of client $i, i \in [1, K]$ over the secret test set $D_i^{val}$ at communication round $t$ is $\nu_i^t$ and $\zeta_i^t$, respectively. The BMTA and BMT-F1 metrics are calculated as: $\text{BMTA} = \max_{{t}} \frac{1}{K} \sum_{i=1}^{K} \nu_i^t$ and $\text{BMT-F1} = \max_{{t}} \frac{1}{K} \sum_{i=1}^{K} \zeta_i^t$.

\subsection{Performance on DP-synthetic data}

Table \ref{tab:main_results} shows that our proposed framework PPPFL outperforms the baselines by a significant margin. Specifically, we outperform the baselines by a significant margin in multiple datasets, we also push the frontier of data privacy and data utility by achieving good performance while maintaining DP guarantees. In terms of accuracy, our proposed method outperforms previous works in all four benchmarks, MNIST, FMNIST, CIFAR-10, and CIFAR-100, with the corresponding figures: 0.9640, 0.9199, 0.80, and 0.480. Regarding BMT-F1, in the CIFAR-10 dataset, our framework is also superior to the FedMeta algorithms by more than 2.5\%, while this figure achieves by nearly 8.0\% regarding the CIFAR-100 dataset.

\begin{table*}[!htbp]
\caption{The average performance over the clients, train on synthetic, fine-tuned and test on secret data}
\resizebox{\0.95 \textwidth}{!}{%
\begin{tabular}{l|cc|cc|cc|cc}
\multicolumn{1}{c|}{} & \multicolumn{2}{c|}{\textbf{MNIST}} & \multicolumn{2}{c|}{\textbf{FMNIST}} & \multicolumn{2}{c|}{\textbf{CIFAR-10}} & \multicolumn{2}{c}{\textbf{CIFAR-100}} \\ \hline
Method & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l}{BMTA} \\ \hline
FedAvg & 0.9223 & 0.9051 & 0.7267 & 0.7938 & 0.4898 & 0.6562 & 0.2142 & 0.3028 \\
FedNova & 0.8773 & 0.8996 & 0.6303 & 0.7607 & 0.3847 & 0.6627 & 0.2855 & 0.3374 \\
FedProx & 0.9006 & 0.9040 & 0.6994 & 0.7721 & 0.4856 & 0.6718 & 0.3014 & 0.3404 \\
SCAFFOLD & 0.8906 & 0.9024 & 0.6698 & 0.8061 & 0.3732 & 0.6614 & 0.2096 & 0.3333 \\
FedMeta & 0.9399 & 0.9616 & 0.8367 & 0.8800 & 0.6897 & 0.7466 & 0.2737 & 0.4133 \\
\textbf{PPPFL} & \textbf{0.9420} & \textbf{0.9640} & \textbf{0.8622} & \textbf{0.9199} & \textbf{0.7163} & \textbf{0.8000} & \textbf{0.3526} & \textbf{0.4800} \\ \hline
\end{tabular}
}
\label{tab:main_results}
\end{table*}

\subsection{Performance on non-private synthetic data}
Table \ref{tab:ablation_non_private_gans} shows that when using only a single global model, in comparison with SDA-FL \cite{li2022federated}, where a set of multiple global models are employed, our proposed framework still achieves comparable performances in MNIST and FMNIST datasets and obtain higher accuracy in CIFAR-100 dataset, at 0.7133.

\begin{table}[]
    \centering
    \caption{The BMTA of adapted models when using non-private GANs}
    \resizebox{0.95 \columnwidth}{!}{%
    \begin{tabular}{l|ccc}
    Method         & \multicolumn{1}{l}{\textbf{MNIST}} & \multicolumn{1}{l}{\textbf{FMNIST}} & \multicolumn{1}{l}{\textbf{CIFAR-10}} \\ \hline
    FedAvg         & 0.9400                                           & 0.7893        & 0.5521               \\
    FedNova        & 0.8386                                          & 0.7782         & 0.5724               \\
    FedProx        & 0.9373                                          & 0.7981           & 0.5737                \\
    SCAFFOLD       & 0.9279                                          & 0.8061           & 0.5608             \\
    SDA-FL         & \textbf{0.9826}                                & \textbf{0.8687}  & 0.6789                       \\
    \textbf{PPPFL}          & 0.9466                                 & 0.8666    & \textbf{0.7133}
    \end{tabular}
}
    \label{tab:ablation_non_private_gans}
\end{table}

\subsection{Performance on other data flows}

In Table \ref{tab:ablation_training_test_setup}, the first two rows show the results of models trained locally, whereas collaboration between these models is illustrated in the last three rows. More specifically, without collaboration, the first row shows the results when training on secret data and testing on secret data, while the second row indicates the utility of synthetic data, which is the DP-guarantee data coming with the trade-off of utility. In addition, in the case of collaborative methods, the performance when we naively use the synthetic data, perform an additional fine-tuning step after emerging from the FL protocol, and use FL on private data are shown respectively in the following three rows.

        \begin{table*}[!htbp]
    \centering
    \label{tab:ablation_training_test_setup}

    \begin{threeparttable}
    \caption{Additional experiments on different training and testing setups\tnote{1}}

    \begin{tabular}{c|l|cc|cc|cc|cc}
    \multirow{2}{*}{\textbf{Method}} & \multicolumn{1}{c|}{\multirow{2}{*}{\textbf{Train / test setup}}} & \multicolumn{2}{c|}{\textbf{MNIST}} & \multicolumn{2}{c|}{\textbf{FMNIST}} & \multicolumn{2}{c|}{\textbf{CIFAR-10}} & \multicolumn{2}{c}{\textbf{CIFAR-100}} \\ \cline{3-10}
     & \multicolumn{1}{c|}{} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l}{BMTA} \\ \hline
    \multirow{2}{*}{\textbf{Local}} & secret / secret & 0.8292 & 0.8412 & 0.6896 & 0.7136 & 0.5959 & 0.6130 & 0.2721 & 0.2998 \\
     & synthetic / secret & 0.3021 & 0.3371 & 0.0716 & 0.0838 & 0.1450 & 0.1578 & 0.0067 & 0.0100 \\ \hline
    \multirow{3}{*}{\textbf{Collaborative}} & synthetic / secret & 0.2689 & 0.3157 & 0.0476 & 0.1231 & 0.0500 & 0.1417 & 0.0089 & 0.0100 \\
     & synthetic - fine-tune / secret\tnote{2} & 0.9223 & 0.9051 & 0.7267 & 0.7938 & 0.5510 & 0.7124 & 0.2142 & 0.3028 \\
     & secret / secret & 0.9360 & 0.9645 & 0.8730 & 0.8972 & 0.7427 & 0.8083 & 0.4520 & 0.5416 \\ \hline
    \end{tabular}

    \begin{tablenotes}
        \item[1] The "Train/test setup" column in the table specifies the data used for training and testing the models for each method. The format of the entries in this column is "train set / test set", where the "train set" is the data used to train the model and the "test set" is the data used to evaluate the model's performance.
        \item[2] refers to a training and testing setup in which the model is first trained on synthetic data, then fine-tuned on real, secret data, and finally evaluated on real, secret data test set.
    \end{tablenotes}

    \end{threeparttable}
    \end{table*}

\subsection{Trade-off between Synthetic Data Privacy Levels and Model Performance}

\begin{table}[]
\caption{Test accuracy and Macro F1-score with respected to synthetic data quality}
\resizebox{0.95 \columnwidth}{!}{%

\begin{tabular}{l|llll}
\hline
                             & \textbf{BMTA} \big\uparrow & \textbf{BMT-F1} \big\uparrow & \textbf{IS} \big\uparrow & \textbf{FID} \big\downarrow \\ \hline
$\epsilon \in [1 \ldots 5]$  & 0.8133                 & 0.7454                & 1.2242          & 407.5517           \\
$\epsilon \in [6 \ldots 10]$ & 0.8400                 & 0.7940                & 1.8100          & 249.0475           \\
$\epsilon = \infty$          & 0.8533                 & 0.8199                & 2.9442          & 49.9617            \\ \hline
\end{tabular}
}
\label{tab:ablation_synthetic_quality}
\end{table}

We evaluate our proposed framework under different users' data privacy budget $\epsilon$, from the low range $\epsilon
\in [1 \ldots 5] $ to a higher range between $[6, \ldots, 10]$, and lastly a scenario where the synthetic data is generated by a generative model without having any added DP-noise. The quality of those aforementioned synthetic data sets is evaluated using Inception Score (IS) and Frechet Inception Distance (FID) \cite{}. The results shown in Table \ref{tab:ablation_synthetic_quality} and Fig. \ref{fig:ablation_synthetic_chart} indicate that the higher DP-noise added in the synthetic data might result in higher generalization error.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\linewidth]{figures/FID.pdf}
    \caption{Test accuracy and FID score with respect to different users' privacy budgets. The left vertical axis represents the BMTA in percentage, the right vertical axis indicates the synthetic data quality in terms of FID score while the categories in the horizontal axis are different ranges of clients' privacy budget $\epsilon$, specifically from $1$ to $5$, $6$ to $10$, and lastly $\texttt{inf.}$ represents synthetic data without any DP guarantees.}
    \label{fig:ablation_synthetic_chart}
\end{figure}

\subsection{User incentive to participate in the protocol}
In this part of our work, we focus on the incentives for users to participate in the PPPFL framework, as shown in Fig. \ref{fig:incentive_analysis}. We demonstrate a strong correlation between the local performance of a user's model and the benefit that the user derives from participating in the protocol, as indicated by a high Spearman's correlation coefficient. In other words, users whose local models have lower performance stand to gain more from participating in the PPPFL protocol.

\begin{figure}
\centering
  \label{fig:incentive_analysis}
  \begin{subfigure}[b]{0.75\columnwidth}
    \includegraphics[width=\linewidth]{figures/incentive_changes.pdf}
    \caption{Accuracy improvements of adapted federated models over local, trained-from-scratch models.}
    \label{fig:incentive_changes}
  \end{subfigure}
  \hspace{0.5 cm}
  \begin{subfigure}[b]{0.75\columnwidth}
    \includegraphics[width=\linewidth]{figures/incentive_changes_correlation.pdf}
    \caption{The correlation between local models' performance with the performance gain from protocol participation.}
    \label{fig:incentive_changes_correlation}
  \end{subfigure}

  \caption{The relationship between the enhancements of adapted federated models over locally trained models. The Spearman's correlation coefficient is  $ -0.90$ indicates a strong correlation between local models' performance with the performance gained by taking part in the protocol.}
\end{figure}

This phenomenon raises the possibility of fairness issues within the PPPFL framework, which we plan to address in future work.

\subsection{Discussion}

\section{Conclusions and Future Works}
\label{sec:discussion}
In this work, we showed that our proposed Personalized Privacy-Preserving Federated Learning (PPPFL) framework could not only minimize the risk of sensitive information of users' data being leaked but also overcome the challenges of the nonindependent and identically distributed (non-IID) data among clients. We demonstrated the efficiency of our proposed framework on different datasets, including MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. Experiment results showed that the PPPFL framework outperforms many existing baselines on both DP-synthetic and non-private synthetic data. We also showed the existence of a trade-off between synthetic data privacy levels and the model's performance. In particular, the higher DP noise added to the synthetic data might result in higher generalization errors. Despite our performance improvements compared to the baselines, conducting experiments with other GAN architectures is necessary.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi

The authors would like to thank...

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\section{Introduction}\label{sec:introduction}}
\else
\section{Introduction}
\label{sec:introduction}
\fi

\IEEEPARstart{T}{his} demo file is intended to serve as a ``starter file''
for IEEE Computer Society journal papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later.
I wish you the best of success.

\hfill mds

\hfill August 26, 2015

\subsection{Subsection Heading Here}
Subsection text here.

\subsubsection{Subsubsection Heading Here}
Subsubsection text here.

\section{Conclusion}
The conclusion goes here.

\appendices
\section{Proof of the First Zonklar Equation}
Appendix one text goes here.

\section{}
Appendix two text goes here.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi

The authors would like to thank...

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\section{Introduction}

\section{Related Works}
Subsection text here.

\subsubsection{Subsubsection Heading Here}
Subsubsection text here.

\section{Conclusion}
The conclusion goes here.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi

The authors would like to thank...

\section{Introduction}\label{sec:introduction}}

\IEEEPARstart{A}{lthough} deep learning (DL) approaches have achieved significant accomplishments in a wide range of applications \cite{suvorov2022resolution, saharia2022photorealistic, schulman2022chatgpt}, there exists several fundamental challenges \cite{dong2021survey} that yet need to be addressed in order to enable its complete power. Due to data sovereignty and data privacy concerns, one major problem is the difficulty in the centralization of the local data from multiple clients to one central server.
In order to guarantee the privacy-preserving in training schemes, federated learning (FL) framework \cite{mcmahan2017communication} is proposed to allow multiple entities (e.g., individuals, or organizations) collaboratively train a DL model under the coordination of a central server. Specifically, each client's data is trained and stored locally, and only updated gradients are transferred to the server for aggregation purposes. By keeping the training data locally and in a decentralized fashion, FL approaches can alleviate many of the systemic privacy risks of traditional centralized DL methods \cite{zhang2021survey}.

However, under FL's naive implementation, there are several key challenges \cite{yang2019federated}, including private information leakage, verifiable computation, expensive communication costs, and statistical heterogeneity.
In the scope of this research, we focus on the problem of private information being divulged in FL settings. Generally, despite the local training without data exchange between the server and clients, private information can still be leaked to some extent from the local gradients through inversion attacks \cite{geiping2020inverting,wang2019beyond,yin2021see,zhao2020idlg,zhu2019deep}. More seriously, as the central server holds full control of observing and modifying the shared global model, sensitive information can also be eavesdropped on by the server. For instance, \citeauthor{boenisch2021curious} \cite{boenisch2021curious} identified that an active-and-dishonest server can set up trap weights for recovering the clients' data with zero reconstruction loss.

With the aim of limiting information disclosure about private data, a state-of-the-art privacy-preserving technique was proposed by \citeauthor{dwork2014algorithmic} \cite{dwork2014algorithmic}, known as differential privacy (DP). The central principle of this technique is to add curated noises to the computational steps to perturb the sensitive information.
In general, there are two approaches to incorporating DP into FL settings: global DP \cite{dwork2006calibrating} and local DP \cite{duchi2013local}. Under global DP, clients are assumed to trust the centralized server, and noises are added by the server; while under local DP, each client perturbs its information locally prior to sharing it with the server without requiring trust in a server \cite{kasiviswanathan2011can}. Despite the addition of noises, global DP approaches cannot ensure privacy preservation in the case of the server being dishonest. Therefore, in this research, we concentrate on the local DP scenario to enhance the privacy protection of the clients against gradient leakage and parameter modification.

Another issue in FL is the non-independent and identically distributed (non-IID) \cite{zhu2021federated, kulkarni2020survey}, caused by the heterogeneity in local data distributions among clients. In particular, in non-IID scenarios, different clients train models from different data distributions, which leads to the performance degradation of the aggregated model. This problem was especially taken into consideration in the cases of cross-silo FL  where clients are large entities (e.g., companies, institutions, and hospitals), and their private data may be greatly different in size and partitioning \cite{kairouz2021advances}. There are plenty of approaches introduced for handling non-IID problems, such as sharing data with the server \cite{tuor2021overcoming}, fine-tuning the local models after obtaining the aggregated model \cite{wang2019federated, t2020personalized}, adding personalized layers in local models \cite{arivazhagan2019federated}, and transferring knowledge from the server to a certain client \cite{liu2020secure, lin2020ensemble}. While those methods mitigate the non-IID data distribution in FL in some cases, they inevitably increase the risk of privacy leakage and a large number of hyperparameters \cite{zhu2021federated}.

\subsection{Our Contributions}

In this work, to mitigate the above challenging FL problems, we propose a Personalized Privacy-Preserving Federated Learning (PPPFL) framework for cross-silo FL, a novel framework that allows local clients simultaneously protect their own initial data at the instance-level and achieve a personalized model. To the best of our knowledge, this study is the first to marry two lines of research in FL on privacy-preserving and non-IID data, to achieve both differential privacy guarantees and better convergence performance. This framework consists of two computational stages: server-side computations and client-side computations. In terms of the former stage, building on meta-learning methods, the primary goal is to find an initial global model shared between all clients which performs well after each client updates it with respect to its own generated artificial data (Section \ref{sec:server-compute}). In this way, the final model implemented by each local client can achieve stronger generalization guarantees and more personalized adaptation. Regarding the latter stage, each local client generates synthetic data by training a differentially private data generative model (Section \ref{sec:dp-training}).
Those data are then utilized for collaboration purposes in the Federated training phase, therefore preventing sharing of the raw data (Section \ref{sec:client-compute}). Moreover, we also focus on cross-silo FL as attention put on applying FL to multiple large organizations has greatly grown \cite{cordis2019machine, treleaven2022federated}. Therefore, it is sufficient for our proposed GANs-based framework to guarantee the DP in cross-silo FL.
\\

We summarize our contributions as follows:

\begin{itemize}
    \item We propose a PPPFL, a novel framework that not only prevents information leakage but also tackles the non-IID issue. To the best of our knowledge, we are the first to introduce a new approach to provide the instance-level privacy-preserving and address data heterogeneity in FL.

    \item We first consolidate several advantages of \textsc{DataLens} into a single FL framework to provide instance-level DP, helping to minimize the risks of sensitive information leakage.

    \item We introduce a new variant of FedMeta in terms of personalization and convergence by incorporating the clients' privacy budgets and gradient stabilization techniques into the optimization process.

    \item We conduct thorough experiments and ablation studies to demonstrate the effectiveness of the proposed approach on different datasets.\\

\end{itemize}
\leavevmode
\noindent  The rest of this paper is organized as follows. Related works on privacy-preserving FL and non-IID challenges in FL are reviewed in Section \ref{sec:related}. In section \ref{sec:preliminary}, we introduce backgrounds on threat models, differential privacy, data generative models, and optimization-based meta-learning methods. Section \ref{sec:method} first presents our problem formulation and details of the proposed method, then performs a rigorous analysis of its privacy guarantee and convergence. Section \ref{sec:evaluation} provides extensive experiments on multiple benchmark datasets. Finally, section \ref{sec:discussion} discusses the experimental results, some key findings, and limitations of this work and concludes the paper.

\section{Related Works}
\label{sec:related}
\noindent
\textbf{Privacy-preserving FL.}
Because of the Active-and-dishonest threat server, in this work, we mainly focus on local DP in which noise addition is conducted by each local client to prevent perfect extractability of individually private information. Many DP approaches based on the SGD algorithm have been proposed to protect the privacy of clients. \citeauthor{sun2020ldp} \cite{sun2020ldp} used the adaptive range of clients' weights to achieve better-aggregated model performance, especially in deeper models, and leveraged parameter shuffling as an extra layer of privacy protection. Meanwhile, \cite{truex2020ldp} allowed the individual clients to define their own local privacy setting by developing a local training approach and a parameter update sharing method, based on Condensed Local Differential Privacy with Exponential Mechanism \cite{gursoy2019secure}. In order to minimize the tradeoff between the privacy budget and the convergence performance, \citeauthor{wei2021user} \cite{wei2021user} proposed a DP-SGD-based communication rounds discounting method.

Another approach to mitigating the privacy concern for data sharing is to utilize generative models. The key idea of this approach is consolidating generative models to locally synthesize data, and then these generated data would be sent to the server by clients for collaboration. \citeauthor{chen2018differentially} \cite{chen2018differentially} first leveraged the power of autoencoder in FL that allows clients to generate their own artificial data by independently feeding their private data into the autoencoder. Under the FL framework and DP thinking, \cite{zhang2021feddpgan} introduced FedDPGAN for COVID-19 pneumonia detection without compromising patients' sensitive information. In this method, each medical institution adds Gaussian noise in training gradients, and then parameters are sent to the server for training COVID-19 diagnostic models collaboratively.

However, those approaches have not taken non-IID data into consideration which inevitably causes a deterioration in the accuracy of FL. In this research, we not only handle privacy concerns but also focus on non-IID problems. Moreover, in real-world industrial environments which is referred to as the cross-silo setting, \cite{mahawaga2022local} utilized RAPPOR \cite{erlingsson2014rappor} and optimized unary encoding that can efficiently control the privacy budgets and address the massive data distribution in many industrial applications (e.g., healthcare, smart cities). Inspired by this work, we also provide a framework for privacy-preserving FL to be utilized under cross-silo settings, potentially with untrusted parties.\\
\leavevmode
\\
\noindent \textbf{Solving the non-IID problem in FL.} Due to the heterogeneity in participants' data distributions, possessing a global model that can fit all the models among the parties is difficult \cite{kairouz2021advances}. To handle this issue, personalization approaches have received much attention recently. The early work \cite{smith2017federated} cast the personalization problem as a multi-task learning problem, which generates separated but associated models for each client. Meanwhile, \cite{arivazhagan2019federated, liang2020think} allowed each client to have personalized layers followed by base layers, and only the base layers are shared with the parameter server. This approach is able not only to boost the learning performance of FL on unknown heterogeneous data but also to reduce the communication costs since clients are not necessarily to share the whole model with the server. Knowledge distillation is also a promising approach for personalized FL with the main idea of transferring knowledge from the server or other clients to a particular client to enhance its performance on non-IID data. Transfer learning is utilized by \cite{liu2020secure, lin2020ensemble}, in which \citeauthor{liu2020secure} incorporated additively homomorphic encryption (HE) and secret sharing while \citeauthor{lin2020ensemble} leveraged an ensemble distillation strategy respectively, to allow knowledge to be shared without compromising user privacy. Another important chart of knowledge distillation is federated domain adaptation which aims at eliminating the differences between data shards between clients \cite{li2019fedmd, peng2019federated}. The most effective and powerful personalization method is to incorporate meta-learning within FL. MAML \cite{finn2017model} has been leveraged in  \cite{fallah2020personalized}, where authors proposed Personalized FedAvg to seek a proper initialization model for the clients that can be easily updated with the local data of each client after the training phase.

Motivated by meta-learning approaches, we also take advantage of MAML to provide personalized models for clients with non-IID data but introduce a stabilized version of MAML that is more easily convergent compared to that in \cite{fallah2020personalized}. Similar to studies \cite{huang2021personalized, shamsian2021personalized}, our work also emphasizes on cross-silo FL to address challenges in real-world settings, such as high coordination and operational cost \cite{kairouz2021advances}.

\section{Preliminaries}
\label{sec:preliminary}
In this section, we provide the details of threat models in privacy when it comes to the existence of an active-and-dishonest server. Next, we provide some background knowledge on DP and the local DP guarantees in FL. Finally, we briefly discuss Optimization-based Meta-learning through the MAML \cite{finn2017model} framework. We also summarize the crucial notations used in this paper in Table \ref{tab:notations}.

\begin{table}[H]
\caption{Main notations used in the paper.}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{l|l}
\hline
$\theta$                        &   Global model's parameters               \\ \hline
$\theta_i$                      &   Local model's parameters in client $i$  \\ \hline
$D_i$                           &   Secret (private) dataset in client $i$            \\ \hline
$\mathcal{G}_i$                 &   Local DP data generator of client $i$ trained from $D$ \\ \hline
${D^*}_i$                &   Synthetic dataset of client $i$ generated by $\mathcal{G}_i$\\ \hline
$\epsilon_i$                    &   Privacy budget chosen by client $i$ when training $\mathcal{G}_i$       \\ \hline
$\alpha^{*}$                    &   Inner learning rate when training local model $\theta$ using synthetic data ${D^*}$   \\ \hline
$\beta^{*}$                     &   Outer learning rate when server aggregate local models $\theta$ \\ \hline
\end{tabular}
}
\label{tab:notations}
\end{table}

\subsection{Threat models}
\textbf{Semi-honest server.} In this type of server, all participated parties follow the protocol specification to perform the computation task but they may try to infer the data of other parties from the received intermediate computation results \cite{li2021trustworthy}.

\citeauthor{geiping2020inverting} indicated that through data recovery attacks \cite{7958568}, the gradients of any fully connected (FC) layers followed by a ReLU activation function $\mathbf{y} = \textsc{ReLU}(W\mathbf{x} + \mathbf{b})$ contain the training data point $\mathbf{x}$ that may result in the possibility of the original data being reconstructed from the knowledge of the parameter gradients.
Formally, when the $i^{th}$ neuron of a FC layer is fired (i.e., $\mathbf{w}_i^T \mathbf{x} + b_i > 0$), the data point $\mathbf{x}$ can be reconstructed by computing $\mathbf{x}^{T}=\left(\frac{\partial \mathcal{L}}{\partial b_{i}}\right)^{-1} .  \left(\frac{\partial \mathcal{L}}{\partial \mathbf{w}_{i}^{T}}\right)$, where $\mathbf{w}_i^T$ be the  $i^{th}$ row in the weight matrix and $b_i$ be the  $i^{th}$ component in the bias vector.
 Consequently, a passive attacker (e.g., semi-honest server) can extract roughly 20\% of arbitrary data points from a batch size $B =  100$ for 1000 neurons on ImageNet \cite{7958568}.\\
\\
\noindent \textbf{Active-and-dishonest server.}
Due to the existence of the data leakage through gradients of the FC layer, \citeauthor{boenisch2021curious} has raised high privacy-related concerns in FL settings as the central party holds full control of observing and modifying the shared gradients. Although the addition of the calibrated noise into the users' data or shared parameters can prevent the data reconstruction attack \cite{wei2020federated, adnan2022federated, ramaswamy2020training}, these implementations provide virtually no protection against a non-honest server since the additional step is performed by the server but not the clients themselves. Therefore, it is evident that for the active-but-dishonest server, no privacy guarantees can be provided to the clients.
In this paper, our goal is to design a differentially private framework, which ensures that the synthetic data instead of the sent parameters are differentially private. By this setting, clients only send the model's parameters trained with DPGAN-by generated data to the server for collaboration; thus differential privacy can be guaranteed even in cases of a non-honest central party.

\subsection{Differential Privacy}
$(\epsilon-\delta)$ Differential Privacy proposed by \citeauthor{dwork2014algorithmic}\cite{dwork2014algorithmic} is currently a strong criterion for privacy preservation of distributed data processing systems. It provides a guarantee that adversaries cannot infer sensitive information about individuals based on the algorithm outputs. Some formal definitions and concepts of types of DP in general and in FL are as follows.

\begin{definition}
    \label{def:dp}
    $(\epsilon-\delta)$ Differential Privacy

    Given a randomized mechanism $M$ and adjacent datasets $D$ and $D'$, $M$ satisfies $(\epsilon, \delta)$-differential privacy, if for all outputs $O$ of $M$ and for all possible sets $S$ of outputs

    \begin{equation*}
    \begin{array}{rl}
        \label{eq:epsilon_delta_dp}
        Pr[M(D) \in S] &\leq \exp(\epsilon) * Pr[M(D') \in S] + \delta.
    \end{array}
    \end{equation*}

    This definition captures the idea that the probability of any particular output should not be significantly affected by the presence or absence of any individual's data in the general dataset. The parameters $\epsilon$ and $\delta$ control the strength of the privacy guarantee, with smaller values of $\epsilon$ and $\delta$ indicating a higher degree of privacy preservation.

\end{definition}

\begin{definition}
    \label{def:rdp}
    $(\lambda, \alpha)$-Renyi Differential Privacy

    A randomized mechanism ${M}$ guarantees $(\lambda, \alpha)$-RDP with $\lambda>1$, if for any neighboring datasets $D$ and $D'$

    \resizebox{0.95\columnwidth}{!}{
    $D_\lambda\left({M}(D) | {M}\left(D^{\prime}\right)\right)=\frac{1}{\lambda-1} \log \mathbb{E} _ {x \sim {M}(D)} \left[\left(\frac{\operatorname{Pr}[{M}(D)=x]}{\operatorname{Pr}\left[{M}\left(D^{\prime}\right)=x\right]}\right)^{\lambda-1}\right] \leq \alpha.$
    }
    \end{definition}

\begin{theorem} (From RDP to DP \cite{mironov2017renyi})

    If a mechanism $\mathcal{M}$ guarantees $(\lambda, \alpha)$-RDP, then it also guarantees $\left(\alpha+\frac{\log 1 / \delta}{\lambda-1}, \delta\right)$-differential privacy for any $\delta \in(0,1)$.

\end{theorem}

\noindent
\textbf{Local DP and Global DP.}
In FL, global differential privacy (global DP) refers to a privacy-preserving technique in which a trusted aggregator adds noise to multiple users' data or parameter updates to protect their privacy. The noise is added using a DP mechanism, which is designed to ensure that the noise is sufficient to prevent the identification of any individual user's data, even when the aggregator has access to the data of all users.
However, when the central server, or aggregator, is active and dishonest, it could identify the data of individual users. Hence, local DP adds noise to the data locally before it is sent to the central server or transmitted over an insecure network.

\textbf{User-level DP and instance-level DP in FL.}
User-level DP is a variant of local DP used when the data consists of a smaller number of samples associated with specific users, and the focus is on protecting the privacy of the users rather than the privacy of the individual data points.

On the other hand, instance-level differential privacy is generally more robust than user-level DP because it provides stronger privacy guarantees for individual data samples. Particularly, instance-level DP ensures that the probability of an adversary learning sensitive information about a specific data point is bounded by a small privacy budget. Moreover, instance-level DP is  easier to implement and apply in practice than user-level DP, as it does not require tracking or identifying specific users or user groups in the dataset, thus making it more flexible and easier to use, especially in the context of cross-silo FL \cite{liu2020learning, wei2021user, levy2021learning}.

\subsection{Differentially Private Data Generation}

Generative Adversarial Networks (GANs) are generative models that consist of a generator and a discriminator, which are trained to play a minimax game to generate synthetic data that is similar to a given dataset.
More formally,  the generator $\mathcal{G}$ maps from a noise variable $z \sim p_z(z)$ to a fake sample $\mathcal{G}(z)$ and to trick the discriminator $\mathcal{D}$ from classifying fake sample against real sample $x \sim p_r(x)$.
The loss function can be formulated as

\begin{equation*}
\label{eq:gan}
\begin{split}
    \min_\mathcal{G} \max_\mathcal{D} L(\mathcal{D}, \mathcal{G})  &= \mathbb{E}_{x \sim p_{r}(x)} [\log \mathcal{D}(x)] \\
    &\quad + \mathbb{E}_{z \sim p_z(z)} [\log(1 - \mathcal{D}(\mathcal{G}(z)))].
\end{split}
\end{equation*}

GANs are successfully utilized in plenty of applications \cite{wang2021generative}, such as image in-painting, image super-resolution, image-to-image translation, language, and speech synthesis.
Due to the high expressiveness of deep neural networks in modeling the data distribution, DL-based generative models can output high-fidelity samples. However, similar to other DL-based architectures, GAN models are also vulnerable to several privacy violation attacks \cite {chen2018differentially} that might extract massive sensitive information from generated data. In order to provide rigorous privacy guarantees to synthetic data, many approaches adopted DP in GANs in hopes of enabling high learning efficacy without disclosing much sensitive information about individuals within the training set. DP-GAN \cite{xie2018differentially} and DP-CGAN \cite{torkzadehmahani2019dp} are two earlier approaches introduced to adapt DP-SGD \cite{abadi2016deep} to GAN. The principle of this method is to add calibrated Gaussian noise to the discriminator gradients during the training step. Later, in order to address a major limitation of prior studies related to privacy budget explosion, \citeauthor{papernot2018scalable} \cite{jordon2018pate} integrated the off-the-shelf Private Aggregation of Teacher Ensembles (PATE) framework \cite{papernot2018scalable} into GAN to provide better privacy budget usage while mitigating the sensitive data disclosure. Specifically, PATE-GAN first learns a set of teacher discriminators on a disjoint subset of training data and then aggregates the output of these teacher models to train a student discriminator. Another variant, G-PATE \cite{long2019scalable} improved PATE-GAN by directly training scalable differentially private student generators to preserve high generated data utility. Nevertheless, those still cannot generate differentially private high dimensional data with strong privacy guarantees. By combining the PATE framework with a $top-k$-based proposed noisy gradient compression and aggregation strategy \textsc{TopAgg}, \textsc{DataLens} \cite{wang2021datalens} demonstrated the capability of generating high-dimensional data with limited privacy budgets and communication-efficient distributed learning, outperforming PATE-GAN and G-PATE.
Taking inspiration from DP-GAN, especially \textsc{DataLens}, we can enjoy the benefits of them to synthesize high-utility data while protecting individual training information. We provide a detailed description in Section \ref{sec:client-compute} and \ref{sec:server-compute}.

\subsection{Optimization-based Meta-learning}

The goal of meta-learning is to train a model that can quickly adapt to a new task, based on only a few examples, through two phases: meta-learning and adaptation. To achieve effectiveness, the meta-learning problem treats entire tasks as training samples. Optimization-based approaches cast the adaption part of the process as an optimization problem \cite{finn2017model}. This  aims at establishing initial model parameters $\theta$ through learning from $T$ number of similar tasks, which can perform well on a new task $T+1$.
Formally, given a task $\mathcal{T}$, and its associated training and validation dataset $(\mathcal{D}^{tr}, \mathcal{D}^{val})$, the optimal parameter $\theta^{*}$ for $\mathcal{D}^{tr}$ is calculated by a gradient descent step
\begin{equation*}
    \theta^{*} = \theta - \alpha\nabla_{\theta}\mathcal{L}(\theta, \mathcal{D}^{tr}),
\end{equation*}
where $\mathcal{L}$ is the loss function, each $\mathcal{D}$ contains observations $X$ and corresponding targets $y$, $\mathcal{D} = \{X, y\}$.
Different from transfer learning, we need to optimize the initial parameter $\theta$ such that it would perform well on the validation set $\mathcal{D}^{val}$. In meta-learning, this step is known as the meta-training step
\begin{equation*}
    \theta = \theta - \beta\nabla_{\theta}\mathcal{L}(\theta^{*}, \mathcal{D}^{val}).
\end{equation*}

For multiple tasks $\mathcal{T} = \{\mathcal{T}_1, \mathcal{T}_2, ..., \mathcal{T}_t\}$, the optimal parameter $\theta^{*}$ for a task $\mathcal{T}_i$ is computed
\begin{equation*}
    \theta^{*}_i = \theta - \alpha\nabla_{\theta}\mathcal{L}(\theta, \mathcal{D}^{tr}_i),
\end{equation*}
with the corresponding updating rule
\begin{equation*}
    \theta = \theta - \beta\nabla_{\theta}\sum_i{\mathcal{L}(\theta^{*}_i, \mathcal{D}^{val}_i)}.
\end{equation*}

A task $t$ contains i.i.d. observations $X_t$ and their corresponding targets $y_t$.
Similarly, to evaluate a meta-learning algorithm, one usually split each task's data $(X_t, y_t)$ into the training (support) set $(X_t^{tr}, y_t^{tr})$ and validation (query) set $(X_t^{val}, y_t^{val})$. If we define $\mathcal{L}_t: \mathbb{R}^d \rightarrow \mathbb{R}$ as the loss corresponding to task $t$.

\textbf{Meta-training} is to learn a model that can adapt to new tasks quickly and efficiently using only a small number of gradient descent steps, without the need for extensive training data. Formally, the training objective function can be formulated as follows

\begin{equation}
    \label{eq:meta_training}
    \resizebox{0.9\columnwidth}{!}{
    $L^{train}(\theta) = \frac{1}{T}\sum_{t=1}^{T}{\mathcal{L} \left[\left(\theta; X_t^{tr}, y_t^{tr} \right); X_t^{val}, y_t^{val} \right].}$
    }
\end{equation}

\textbf{Meta-testing} is the process of evaluating the performance of a model that has been meta-trained on a set of new tasks. During meta-testing, the model is presented with a novel task that is similar but has not been seen during meta-training. The empirical risk on novel task $T+1$ can be formulated as follows

\begin{equation}
    \label{eq:meta_test}
    \begin{split}
     L^{test}(\theta) &= \mathbb{E}_{(X_{T+1}, y_{T+1}, (x', y') \sim p_{T+1})} \\
     &\quad \mathcal{L}\left[(\theta; X_{T+1}, y_{T+1}); x', y') \right].
     \end{split}
\end{equation}

\begin{figure*}[!ht]
    \centering
    \includegraphics[width=0.75\textwidth]{figures/PPPFL_new.pdf}

    \caption{The overall architecture of the PPPFL framework consisting three consecutive stages. First, each participant locally synthesizes DP-guaranteed dataset ${D^*}$ by training their own DP-data-generator $\mathcal{G}$ from secret dataset $D$. Next, in the federated training process, the synthetic data, which have been respectively separated into a training set ${D^*}^{tr}$ and held-out validation set ${D^*}^{val}$, will be used to update the global model parameters $\theta$. Lastly, the clients get emerged from the federated training rounds and perform updates to their local model parameters $\theta^{*}$ based on their secret training data $D^{tr}$ and evaluate their new locally adapted model with respect to the secret validation set $D^{val}$.}
    \label{fig:framework}
\end{figure*}

\section{Methodology}
\label{sec:method}

In this section, we present a formal problem formulation in which the goal is to enhance privacy in personalized Federated Learning. We then proceed to describe our proposed solution to this problem, the PPPFL framework, in detail. This includes a step-by-step description of the process, beginning with the training of local models on the client side and culminating in the federated collaboration between the clients and the server. Finally, we provide theoretical proof demonstrating the privacy and convergence guarantees of the PPPFL framework.

\subsection{Problem Formulation}
\noindent
\textbf{Privacy-preserving FL.} Given a set of $K$ clients $\{ c_1, \dots, c_K \}$ sampled from a client distribution $\mathcal{C}$ and an un-trusted server $S$.
Each $i^{th}$ client has its own private dataset $D_i$ and all clients' datasets are non-IID. In this research, we mainly study the common non-IID scenario where the local data distribution $\Pr(x_i, y_i) = \Pr(x_i \mid y_i) \Pr(y_i)$ with the label distribution $\Pr(y_i)$ is different among clients.
For the evaluation purpose, local private datasets $D_i$ are divided into training $D_i^{tr}$ and testing $D_i^{val}$ sets, such that $D_i = D_i^{tr} \cup D_i^{val}$. Due to the concern of the active-and-dishonest server $S$, the private datasets will be kept locally and never be used for the federated training phase. Instead, clients train their own personalized  synthetic data generator $\mathcal{G}_i$, and then use the generated datasets ${D^*}_i^{tr}$ and ${D^*}_i^{val}$ with ${D^*}_i = \mathcal{G}_i(D_i)$ for collaboration. The privacy budget $\epsilon_i$ in each $\mathcal{G}_i$ is customized by the clients.

\noindent
\textbf{Personalized FL.} The goal is to find an initial shared model that can perform well after each client updates it with respect to its own gradients. Specifically, the central party uses synthetic datasets ${D^*}_i$ to meta-learn a generalized model parameterized by $\theta$ from the updates sent by the clients.  Therefore, this model can be later personalized by each client with its own corresponding private data $D_i$.  The support set ${D^*}_i^{tr}$ and a query set ${D^*}_i^{val}$, and the meta-model will be locally adapted to the private data set $D_i^{tr}$ and evaluated on $D_i^{test}$ by the clients.

Considering the meta-train tasks and meta-test tasks $t \sim \mathcal{T}$ are now the clients $i \sim \mathcal{C}$ with the synthetic ${D^*}_i$ and private data $D_i$, respectively. The meta-training objective function in Eq. \ref{eq:meta_training} can be re-written as

\begin{equation}
    L(\theta) = \mathbb{E}_{c \sim \mathcal{C}}\mathcal{L}\left[\left({\theta}; {D^*}_{c}^{tr}\right); {D^*}_i^{val}\right].
\end{equation}

\subsection{Our Proposed Framework}
We first introduce the data flow in each client and the central server. Then, we present our techniques to overcome the challenges of gradients instability \cite{antoniou2018train} when using MAML \cite{finn2017model} as the optimization-based meta-learning algorithm, and the computational steps of the \textsc{DataLens} \cite{wang2021datalens} architecture.

\begin{algorithm}[H]
    \caption{\textsc{Computational steps on the clients}}
    \label{alg:clients}
    \begin{algorithmic}[1]
    \renewcommand{\algorithmicrequire}{\textbf{Input:}}
    \renewcommand{\algorithmicensure}{\textbf{Output:}}
    \renewcommand{\algorithmiccomment}[1]{\hfill// #1}
    \Require client data $D$, client model $\theta$, fine-tuning steps $\gamma$
    \Ensure  adapted model $\hat{\theta}$
    \\ \textit{// Before collaborative training}
    \For{each client $c \in \mathrm{C}$}
        \State $\mathcal{G}_{c} \gets \textsc{DP-GAN Training}(D_{c}$) \ref{alg:datalens} \Comment{Training the DP data generator}
        \State (i) $D_{c} \gets D^{tr}_{c} \cup D^{val}_{c}$ \Comment{Locally data partitioning}
        \State (ii) ${D^*}^{tr}_{c} \gets \mathcal{G}_{c}(D^{tr}_{c}),   {D^*}^{val}_{c} \gets \mathcal{G}_{c}(D^{val}_{c})$ \Comment{Synthesizing DP data}
    \EndFor
    \State \textsc{Federated Training} \ref{alg:federatedTraining}

    \textit{// After collaborative training}
    \For{each client $c \in Q$ }
        \State (iii) $\hat{\theta}_{c} \gets \theta_{c} - \gamma \ast\nabla_{\theta_{c}}\mathcal{L}_{c}$

        $\mathcal{L}_{D^{tr}_{c}}(\theta_{c}) \gets \frac{1}{\mid D^{tr}_{c} \mid}\sum_{\left(x,y\right)\in D^{tr}_{c}}{\ell(f_{\theta_{c}}\left(x\right),\ y)}$ \Comment{Locally fine-tuning}
    \EndFor
    \end{algorithmic}
\end{algorithm}

\subsubsection{Method Overview}
We now briefly illustrate the proposed PPPFL framework, which includes 3 main phases: local data generation, federated training, and local adaptation (as depicted in Fig. \ref{fig:framework}). In our method, the DP gradient compression and aggregation approach and the optimization-based meta-learning algorithm serve as key components which help the PPPFL prevent individual information disclosures and improve the consistency among local models.

In the first stage, to limit sensitive information leakage, each client generates its own artificial data through \textsc{DataLens}. \textsc{DataLens} \cite{wang2021datalens} is a state-of-the-art privacy-preserving data generative model, which is capable of generating high-dimensional data without leaking private information. Motivated by this novel DP-GAN model, our framework integrates it to enable each client to add noises to their own data through a training approach (Section \ref{sec:dp-training}). After this data generation step, the synthetic data ${D^*}_i$ of each $i^{th}$ client is used for the federated training, which helps to protect the original data $D_i$ as $D_i$ is kept locally by each client.

\begin{algorithm}[H]
    \caption{\textsc{Aggregations On The Server}}
    \label{alg:server}
    \begin{algorithmic}[1]
    \renewcommand{\algorithmicrequire}{\textbf{Input:}}
    \renewcommand{\algorithmicensure}{\textbf{Output:}}
    \Require initial model $\theta_0$, outer learning rate $\beta^*$
    \Ensure  aggregated model $\theta$
    \State Initialize global model $\theta \gets \theta_0$
    \For{each communication round $t=1,2,\dots$}
        \State Sample a set $U_t$ of $m$ clients and distribute $\theta_0$ for the sampled clients
        \For{each client $c \in U_t$ in parallel}
            \State $g_i \gets \textsc{Federated Training} (\theta_i)$ \ref{alg:federatedTraining}
            \State $\theta \gets \theta - \frac{\beta^*}{m}\sum_{c_t}g_i$
            \State Update the global model
        \EndFor
    \EndFor
\end{algorithmic}
\end{algorithm}

Next, during the Federated training phase (as shown in Algorithm \ref{alg:federatedTraining}), at the beginning of each communication round between clients and the server, each client downloads the weights of a global model. For every following communication round, the clients perform gradient decent updates on the synthetic training data ${D^*}^{tr}$. For the inference step, clients will get feedback by calculating the gradients on the synthetic validation dataset ${D^*}^{val}$ with respect to their own local models, and then send those feedback signals to the server for collaboration (as shown in Algorithm \ref{alg:clients}). Based on those shared signals and clients' models, the global model is updated by the server (as shown in Algorithm \ref{alg:server}).

After the collaboration rounds stop, the clients locally conduct the adaptation phase by performing gradient descent updates on their latest model downloaded from the server with respect to the local private datasets (as shown in Algorithm \ref{alg:clients}) to get better inference performance.

\begin{algorithm}[H]
    \caption{\textsc{Federated Training}}
    \label{alg:federatedTraining}
    \begin{algorithmic}[1]
        \renewcommand{\algorithmicrequire}{\textbf{Input:}}
        \renewcommand{\algorithmicensure}{\textbf{Output:}}
        \Require client synthetic data ${D^*}_i^{tr}, {D^*}_i^{val}$, initial model $\theta_0$, inner learning rate $\alpha^*$
        \Ensure  updated model $\theta^*$
        \State Initialize the local model $\theta \gets \theta_0$
        \For {each communication round $t=1,2,\dots$}
            \State \textit{ClientUpdate($\theta_i$, ${D^*}_i^{tr}$)}
            \State $\mathcal{L}_{D^*_{c}}^{val}(\theta_{c}) \gets \frac{1}{|D^{*val}_{c}|} \sum_{( x^{\prime}, y^{\prime})\in D^{*val}_c}{\ell(f_{\theta_i}(x^{\prime}),y^{\prime})}$
            \State Compute the query loss
            \State $g_i  \gets \nabla_{\theta_{i}} \mathcal{L}_{{D^*}_i^{val}} \left(\theta_i\right)$
            \State Uploading $g_i$ to the server
        \EndFor
        \Function{ClientUpdate}{$\theta_i$, ${D^*}_i^{tr}$}
            \State $\mathcal{L}_{{D^*}_{c}}^{tr}\left(\theta_{c}\right)\gets\frac{1}{\left|{D^*}_{c}^{tr}\right|}\sum_{ \left( x,y \right) \in {D^*}_{c}^{tr}} \ell\left(f_{\theta_i}\left(x\right),y\right)$
            \State $\theta_i  \gets \theta_i - \alpha^* * \nabla_{\theta_i} \mathcal{L}_{D^{*tr}_i}$
        \EndFunction
    \end{algorithmic}
\end{algorithm}

\subsubsection{Client-side computations}
\label{sec:client-compute}

In this framework, besides the gradient sharing process with the server, the clients need to compute two extra steps, which are described in Algorithm \ref{alg:clients}. First, to ensure the instance-level DP for the framework, each client needs to train a differentially private data generator to synthesize their own dataset which is then used for collaboration. Under this setting, our framework can provide stronger privacy guarantees for individual data samples, mitigating information disclosure problems. The second additional step is that clients are expected to perform a local adaptation step on their kept private data after receiving the well-initialized parameters from the central party. This step contributes to the generation of a generalized and personalized model for each client. We assume two added steps are trivial as the clients are large organizations with high computational power.

\subsubsection{Server-side computations}
\label{sec:server-compute}
Different from other FL settings \cite{mcmahan2017communication}, instead of naively aggregating parameters sent by clients, the objective of the server in our setting is to seek the best initialization which performs greatly after each client updates with respect to its own local private dataset. To accomplish this goal, the server aggregates shared parameters and feedback signals sent by clients $(\theta, g)$, where $g$ is the feedback signals and $\theta$ are clients' parameters. However, there always exists a trade-off between the privacy budget and utility, in which higher $\epsilon$ implies higher utility but lower privacy and vice versa. Moreover, as the $\epsilon$ value is controlled by each client, this trade-off is amplified due to the difference in clients' settings. Therefore, it would be more efficient if the server can integrate clients' privacy budget information into its optimization procedure. In other words, the objective function in PPPFL when having multiple $\epsilon$ as

\begin{equation}
    \label{eq:pppfl}
    L(\theta) = \mathbb{E}_{c \sim \mathcal{C}}\left[\frac{e^{\frac{\epsilon_{c}}{\rho}}}{\sum_{j=1}^{K} \frac{e^{\epsilon_{j}}}{\rho}} \mathcal{L}\left(\left({\theta}; {D^*}_{c}^{tr}\right); {D^*}_i^{val} \right)\right],
\end{equation}
with $\rho$ is a hyperparameter that denotes the temperature of the softmax smoothing function over the clients' budgets.

\begin{algorithm}[H]
    \caption{\textsc{DP-GANs Training} }
    \label{alg:datalens}
    \begin{algorithmic}[1]
    \renewcommand{\algorithmicrequire}{\textbf{Input:}}
    \renewcommand{\algorithmicensure}{\textbf{Output:}}
    \Require client private dataset $D_i$, $N$ discriminators $\mathcal{D}_1, \ldots, \mathcal{D}_N$
    \Ensure DP generator $\mathcal{G}$
    \State Partitioning the sensitive dataset $D_i$ into non-overlapping subsets $\mathrm{d}_{1}, \mathrm{~d}_{2}, \ldots, \mathrm{d}_{\mathrm{N}}$ an associate a teacher discriminator $\mathcal{G}_{i}$ to each set.
    \State Training the teachers: using the discrimination loss $\mathcal{L}_{\mathcal{G}}=-\log \mathcal{G}(x)-\log (1 -  \mathcal{G}\left(\mathcal{D}^{\prime}(z)\right)$ with $\mathcal{D}$ denotes the student generator $\mathrm{x}$ is sampled from the subsets $\mathrm{d}_{\mathrm{i}}, z$ is the noise sample.
    \State Gradients Compression: the teacher's dense gradients vectors $g^{(i)}$ into sparsified vectors $\tilde{g}^{(i)}$ with $\{-1,0,1\}$ values.
    \State DP Aggregating and Thresholding: the aggregated teacher's gradients vector $\tilde{g}^* \leftarrow \sum_{i=1}^N \tilde{g}^{(i)}+\mathcal{N}\left(0, \sigma^2\right)$, then be mapped to $\{-1,0,1\}$ values by comparing each gradient direction with a threshold $\beta N$.
    \State Training the student generator: with the generator loss $\tilde{\mathcal{L}}_{\mathcal{G}}(z, \hat{x})=$ $\mathbb{E}\left[\mathcal{D}\left(z \right)-\hat{x}\right]^{2}$, where $\hat{x}=\mathcal{D}\left(\mathrm{z}\right)+\gamma \bar{g}$ is the synthetic data plus the aggregated DP teacher gradients vector and $\gamma$ denotes the learning rate.
\end{algorithmic}
\end{algorithm}

\begin{figure}[ht]
  \begin{subfigure}[b]{0.9\columnwidth}
    \includegraphics[width=\linewidth]{figures/synthetic.pdf}
    \caption{The diverging learning curve.}
    \label{fig:maml_before}
  \end{subfigure}
  \hspace{0.5 cm}
  \begin{subfigure}[b]{0.9\columnwidth}
    \includegraphics[width=\linewidth]{figures/synthetic_after.pdf}
    \caption{The stabilized learning curve.}
    \label{fig:maml_after}
  \end{subfigure}
  \caption{The improvement on convergence (bottom) compared to the original MAML (top) indicated by the total cross-entropy loss of the support set and query set over communication rounds. The top figure indicates a diverging learning curve when the gradients keep accumulating while the bottom figure illustrates fast convergence after 10 communication rounds. The experiment is conducted on the CIFAR-10 dataset.}
\end{figure}

\subsubsection{DP-GAN training}
\label{sec:dp-training}
In this paper, we consolidate the advantages of \textsc{DataLens} \cite{wang2021datalens}, which is a current state-of-the-art DP data generative model, to enable clients to generate their differential privacy dataset. Based on the PATE framework, \textsc{DataLens} consists of multiple teacher discriminators $\mathcal{D}_1, \mathcal{D}_2, \ldots, \mathcal{D}_N$ that produce gradient vectors to guide the student generator $\mathcal{G}$ in updating its synthetic samples. Meanwhile, the student generator sends updated synthetic ones to the teacher for label querying. The curated Gaussian noise $\mathcal{N}(0, \sigma)$ is added into the computations during the teacher aggregation step, which is described in Algorithm \ref{alg:datalens} and \ref{sec:privacy_analysis}.

The detailed training procedure includes five main steps:

\begin{itemize}
    \item \textbf{Step 1: Partitioning the sensitive dataset $D_i$ into non-overlapping subsets.} This technique ensures that the discriminators do not rely on the specifics of a single sensitive training data point, hence protecting the privacy of the training data \cite{papernot2016semi}.
    \item \textbf{Step 2: Training teacher discriminators.} Each teacher discriminator $\mathcal{D}_i$ is trained to differentiate between real samples and synthetic samples generated by the student generator $\mathcal{G}$ by using the loss function $\mathcal{L}_{\mathcal{G}}=-\log \mathcal{G}(x)-\log (1 -  \mathcal{G}\left(\mathcal{D}^{\prime}(z)\right)$.
    \item \textbf{Step 3: Gradients Compression.} The gradient vector of each teacher is filtered by the ${top-k}$ values, then clipped, normalized, and lossy compressed with stochastic sign gradient quantization to reduce communication cost and to improve privacy and convergence guarantees.
    \item \textbf{Step 4: DP gradient aggregation and thresholding.} The Gaussian noise is calibrated by estimating $\ell_2$ sensitivity of the aggregated gradient vector (as shown in \ref{sec:privacy_analysis}). The
    \item \textbf{Step 5: Training the student generator.} When gradient descent steps are performed on the loss function of the student generator, the aggregated DP gradient vectors from the teacher discriminators are backpropagated to the student generator.
\end{itemize}

\subsubsection{Stabilizing MAML algorithm}
As previously noted by \citeauthor{antoniou2018train} \cite{antoniou2018train}, the optimization process in Model-Agnostic Meta-Learning (MAML) can be unstable due to the multiple times that differentiation is performed through the model. This instability is illustrated in Fig. \ref{fig:maml_before}, which shows that the original MAML model diverges after just a few communication rounds. To address this issue, we propose a modified version of MAML that employs Model Exponential Moving Average in the inner loop, followed by Cosine Annealing Learning rate scheduling in the outer loop. This approach is inspired by previous work on improving generalization in DL over Stochastic Gradient Descent (SGD) in general \cite{izmailov2018averaging, tarvainen2017mean, huang2017snapshot} and improving stability in MAML in specific \cite{antoniou2018train}.
Formally, the model parameters $\theta_t$ of the clients at communication round $t$ are set to be an Exponential Moving Average (EMA) of the previous round's model parameters: $\theta_t := \zeta \theta_t + (1-\zeta) \theta_{t-1}$, where $\zeta$ is the momentum hyperparameter.
The learning rate $\beta^{*}$ for the outer optimization loop is calculated as $\beta^*(t)=f(\bmod (t-1,\lceil T / M\rceil))$, where $t$ is the current round of the total $T$ communication rounds, $M$ is the number of cosine annealing cycles, and $f$ is a monotonically decreasing function. By combining these techniques, the global model is able to traverse multiple local optima while maintaining stability during training.

\subsubsection{Personalization}

In our approach, we also take the customization and personalization of clients into consideration since they may have different business-related requirements for data sharing. Instead of giving the server full control on the level of user privacy, we allow clients to adjust their own privacy level depending on their actual demands. This enables clients to independently, and effectively select their preferred privacy budgets $\epsilon$ and incorporate them directly into the DP-GAN training procedure while still being able to control the outcomes of the training process without having to wait for the collaborative model to converge. Additionally, the information about the client's privacy budgets is used in our proposed framework to improve the performance of the client's models.

\subsection{Privacy Analysis
}
\label{sec:privacy_analysis}
In our chosen DP-generative algorithm \cite{wang2021datalens}, briefly described in Algorithm \ref{alg:datalens}, the gradients of the student generator are obtained by aggregating the sparsified gradients of the teachers and voting. Specifically, the gradient set $G=\{g^{(1)},\ldots,g^{(\mathrm{N})} \}$ of $N$ is quantized using top-$k$-sparsification \cite{shi2019understanding} to convert the real-valued gradient vector into a sparse $\{-1, 1\}$ vector $\tilde{g}^{(i)}$ with $k$ non-zero entries.

The Gaussian mechanism can be applied to the sum aggregation function defined as $f_{\text{sum}}(\tilde{\mathcal{G}})=\sum_{i=1}^N \tilde{\mathbf{g}}^{(i)}$ for a set of compressed teacher gradient vectors $\tilde{G}=\left(\tilde{\mathbf{g}}^{(1)}, \ldots, \tilde{\mathbf{g}}^{(N)}\right)$, where $\tilde{\mathbf{g}}^{(i)}$ is the compressed gradient of the $i$-th teacher. The resulting output is $\tilde{\mathbf{G}}\sigma f{\text{sum}}(\tilde{\mathcal{G}})=f_{\text{sum}}(\tilde{\mathcal{G}})+\mathcal{N}\left(0, \sigma^2\right)=\sum_{\tilde{\mathbf{g}} \in \tilde{\mathcal{G}}} \tilde{\mathbf{g}}+\mathcal{N}\left(0, \sigma^2\right)$. The Gaussian mechanism guarantees $(\lambda, \alpha)$-RDP \ref{def:rdp} for any real-valued function $f$.

\begin{lemma} For any neighboring top- $k$ gradient vector sets $\tilde{\mathcal{G}}, \tilde{\mathcal{G}}^{\prime}$ differing by the gradient vector of one teacher, the $\ell_2$ sensitivity for $f_{\text {sum }}$ is $2 \sqrt{k}$.
\end{lemma}

According to Theorem 1, the Gaussian mechanism $\mathrm{G}\sigma f$ to a function $f$ with $\ell_2$ sensitivity results in a $(\lambda, s^2 \lambda /(2 \sigma^2))$-RDP guarantee.
In order to determine the RDP guarantee for the Gaussian mechanism applied to the sum aggregation function $\tilde{\mathrm{G}}\sigma f_{\mathrm{sum}}(\tilde{\mathcal{G}})$, it is necessary to calculate the $\ell_2$ sensitivity of this function. According to Lemma 1, the $\ell_2$ sensitivity of $f_{\text{sum}}$ for any neighboring top-$k$ gradient vector sets $\tilde{\mathcal{G}}, \tilde{\mathcal{G}}^{\prime}$ that differ by a single gradient vector is equal to $2 \sqrt{k}$. This value can be used to determine the RDP guarantee for the Gaussian mechanism applied to the sum aggregation function.

\begin{table*}[ht]
\centering
\caption{The average performance over the clients, trained on synthetic, fine-tuned and test on secret data.}
    \begin{tabular}{l|cc|cc|cc|cc}
    \multicolumn{1}{c|}{} & \multicolumn{2}{c|}{\textbf{MNIST}} & \multicolumn{2}{c|}{\textbf{FMNIST}} & \multicolumn{2}{c|}{\textbf{CIFAR-10}} & \multicolumn{2}{c}{\textbf{CIFAR-100}} \\ \hline
    Method & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l}{BMTA} \\ \hline
    FedAvg \cite{mcmahan2017communication} & 0.9223 & 0.9051 & 0.7267 & 0.7938 & 0.4898 & 0.6562 & 0.2142 & 0.3028 \\
    FedNova \cite{wang2020tackling} & 0.8773 & 0.8996 & 0.6303 & 0.7607 & 0.3847 & 0.6627 & 0.2855 & 0.3374 \\
    FedProx \cite{li2020federated} & 0.9006 & 0.9040 & 0.6994 & 0.7721 & 0.4856 & 0.6718 & 0.3014 & 0.3404 \\
    SCAFFOLD \cite{karimireddy2020scaffold} & 0.8906 & 0.9024 & 0.6698 & 0.8061 & 0.3732 & 0.6614 & 0.2096 & 0.3333 \\
    FedMeta \cite{chen2018federated} & 0.9399 & 0.9616 & 0.8367 & 0.8800 & 0.6897 & 0.7466 & 0.2737 & 0.4133 \\
    \textbf{PPPFL (ours)} & \textbf{0.9420} & \textbf{0.9640} & \textbf{0.8622} & \textbf{0.9199} & \textbf{0.7163} & \textbf{0.8000} & \textbf{0.3526} & \textbf{0.4800} \\ \hline
    \end{tabular}%

\label{tab:main_results}
\end{table*}

\section{Evaluation}
\label{sec:evaluation}
In this section, we illustrate the experimental evaluation of our proposed framework for two problems: prevent sensitive information leakage and address clients' data heterogeneity issues. To demonstrate its effectiveness, we conduct experiments on different benchmark datasets and compare them with six baseline models.

\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/FID.pdf}
    \caption{Test accuracy and FID score with respect to different users' privacy budgets. The left vertical axis represents the BMTA in percentage, the right vertical axis indicates the synthetic data quality in terms of FID score while the categories in the horizontal axis are different ranges of clients' privacy budget $\epsilon$, specifically from $1$ to $5$, $6$ to $10$, and lastly $\texttt{inf.}$ represents synthetic data without any DP guarantees.}
    \label{fig:ablation_synthetic_chart}
\end{figure}

\subsection{Experiment Setup}
\noindent

\begin{table}[ht]
    \centering
    \caption{The BMTA of adapted models when using non-private GANs}
    \resizebox{0.95 \columnwidth}{!}{%
    \begin{tabular}{l|c|c|c}
    Method         & \textbf{MNIST} & \textbf{FMNIST} & \textbf{CIFAR-10} \\ \hline
    FedAvg \cite{mcmahan2017communication}        & 0.9400                                           & 0.7893        & 0.5521               \\
    FedNova \cite{wang2020tackling}       & 0.8386                                          & 0.7782         & 0.5724               \\
    FedProx \cite{li2020federated}       & 0.9373                                          & 0.7981           & 0.5737                \\
    SCAFFOLD \cite{karimireddy2020scaffold}      & 0.9279                                          & 0.8061           & 0.5608             \\
    SDA-FL \cite{huang2021personalized}     & \textbf{0.9826}                                & \textbf{0.8687}  & 0.6789                       \\
    \textbf{PPPFL (ours)}          & 0.9466                                 & 0.8666    & \textbf{0.7133}   \\
    \hline
    \end{tabular}
}
    \label{tab:ablation_non_private_gans}
\end{table}
\noindent
\textbf{Datasets}. We use four public benchmark datasets, including MNIST \cite{deng2012mnist}, Fashion-MNIST \cite{xiao2017fashion}, CIFAR-10 \cite{krizhevsky2010convolutional}, and CIFAR-100, to evaluate our proposed method. MNIST and Fashion-MNIST have $60000$ training examples and $10000$ testing examples, and all of them are grayscale images of $28 \times 28$ dimensions. We also utilize two color-image datasets, CIFAR-10 and CIFAR-100, to demonstrate the performance of the proposed method. Both datasets have $50000$ in training images and $10000$ in testing images.

\noindent
\textbf{Baselines}. We compare our proposed method with the most popular FL algorithm FedAvg \cite{mcmahan2017communication}, and state-of-the-art FL algorithms under non-IID data, including FedNova \cite{wang2020tackling}, FedProx \cite{li2020federated}, SCAFFOLD \cite{karimireddy2020scaffold}, FedMeta \cite{chen2018federated}, and SDA-FL \cite{huang2021personalized}.

\noindent
\textbf{Federated Learning Settings.} We simulate the FL scenario with $5$ clients and $30$ communication rounds. In each communication round, all of the clients participate in the protocol. The inner learning rate and outer learning rate for the meta-training procedure are respectively set to $1e-4$ and $3e-4$, the used optimizer is Adam \cite{kingma2014adam} and batch size is $64$.

\begin{figure}[!htbp]
\centering
  \label{fig:incentive_analysis}
  \begin{subfigure}[b]{0.85\columnwidth}
    \includegraphics[width=\linewidth]{figures/incentive_changes.pdf}
    \caption{Accuracy improvements of adapted federated models over local, trained-from-scratch models.}
    \label{fig:incentive_changes}
  \end{subfigure}
  \hspace{0.5 cm}
  \begin{subfigure}[b]{0.85\columnwidth}
    \includegraphics[width=\linewidth]{figures/incentive_changes_correlation.pdf}
    \caption{The correlation between local models' performance with the performance gain from protocol participation.}
    \label{fig:incentive_changes_correlation}
  \end{subfigure}
  \caption{The relationship between the enhancement of adapted federated models and locally trained models. The Spearman's correlation coefficient is  $ -0.90$ indicating a strong correlation between local models' performance with the performance gained by taking part in the protocol.}
\end{figure}

\begin{table*}[ht]
\begin{threeparttable}
\caption{Additional experiments on different training and testing setups\tnote{1}}
\label{tab:ablation_training_test_setup}
\begin{tabular}{c|l|cc|cc|cc|cc}
\multirow{2}{*}{\textbf{Method}} & \multicolumn{1}{c|}{\multirow{2}{*}{\textbf{Train / test setup}}} & \multicolumn{2}{c|}{\textbf{MNIST}} & \multicolumn{2}{c|}{\textbf{FMNIST}} & \multicolumn{2}{c|}{\textbf{CIFAR-10}} & \multicolumn{2}{c}{\textbf{CIFAR-100}} \\ \cline{3-10}
 & \multicolumn{1}{c|}{} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l|}{BMTA} & \multicolumn{1}{l}{BMT-F1} & \multicolumn{1}{l}{BMTA} \\ \hline
\multirow{2}{*}{\textbf{Local}} & secret / secret & 0.8292 & 0.8412 & 0.6896 & 0.7136 & 0.5959 & 0.6130 & 0.2721 & 0.2998 \\
 & synthetic / secret & 0.3021 & 0.3371 & 0.0716 & 0.0838 & 0.1450 & 0.1578 & 0.0067 & 0.0100 \\ \hline
\multirow{3}{*}{\textbf{Collaborative}} & synthetic / secret & 0.2689 & 0.3157 & 0.0476 & 0.1231 & 0.0500 & 0.1417 & 0.0089 & 0.0100 \\
 & synthetic - fine-tune / secret\tnote{2} & 0.9223 & 0.9051 & 0.7267 & 0.7938 & 0.5510 & 0.7124 & 0.2142 & 0.3028 \\
 & secret / secret & 0.9360 & 0.9645 & 0.8730 & 0.8972 & 0.7427 & 0.8083 & 0.4520 & 0.5416 \\ \hline
\end{tabular}
\begin{tablenotes}
    \item[1] The "Train/test setup" column in the table specifies the data used for training and testing the models for each method. The format of the entries in this column is "train set/test set", where the "train set" is the data used to train the model and the "test set" is the data used to evaluate the model's performance.
    \item[2] refers to a training and testing setup in which the model is first trained on synthetic data, then fine-tuned on real, secret data, and finally evaluated on a real, secret test set.
\end{tablenotes}
\end{threeparttable}
\end{table*}

\subsection{Performance on DP-synthetic data}

\noindent
\textbf{DP data generation model.}
We follow the hyperparameter choices by \cite{wang2021datalens}. Specifically, for the MNIST Handwritten Dataset, the DP GANs are trained using privacy budget $\epsilon \in { 1, \dots, 10} $. The sensitivity $\delta$ is set to $1e-5$, $top-k$ value equals to $300$. The noise parameter $\sigma = 5000$ and voting threshold $\beta = 0.7$.
\\
\noindent
\textbf{Non-IID simulation.}
We simulate the common non-IID scenario where the label distribution is skewed. The label distribution $q \sim Dirichlet(\tau)$ with $\tau = 0.5$.
\\
\noindent
\textbf{Metrics.}
\noindent
We follow the work by \citeauthor{huang2021personalized} \cite{huang2021personalized} on employing Best Mean Test Accuracy (BMTA) as the metric by averaging the testing accuracy on all clients and selecting the best value across the training communication rounds. In cases of the presence of non-IID data, we use the Best Mean Test F1-score (BMT-F1) as an additional metric for better evaluation. Formally, at communication round $t$, given the accuracy score $\nu_i^t$ and the F1-score $\zeta_i^t$ of the $o^{th}$ client ($i \in [1, K]$), the BMTA and BMT-F1 metrics are calculated as $\text{BMTA} = \max_{{t}} \frac{1}{K} \sum_{i=1}^{K} \nu_i^t$ and $\text{BMT-F1} = \max_{{t}} \frac{1}{K} \sum_{i=1}^{K} \zeta_i^t$.

Table \ref{tab:main_results} shows that our proposed framework PPPFL outperforms the baselines by a significant margin. Specifically, we outperform the baselines by a significant margin in multiple datasets, we also push the frontier of data privacy and data utility by achieving good performance while maintaining DP guarantees. In terms of accuracy, our proposed method outperforms previous works in all four benchmarks, MNIST, FMNIST, CIFAR-10, and CIFAR-100, with the corresponding figures: 0.9640, 0.9199, 0.80, and 0.480. Regarding BMT-F1, in the CIFAR-10 dataset, our framework is also superior to the FedMeta algorithms by more than 2.5\%, while this figure achieves by nearly 8.0\% regarding the CIFAR-100 dataset.

\subsection{Performance on non-private synthetic data.}
Table \ref{tab:ablation_non_private_gans} shows that when using only a single global model, in comparison with the ensemble of multiple global models in SDA-FL \cite{li2022federated}, our proposed framework can achieve comparable performances on MNIST and FMNIST datasets and obtain higher accuracy on CIFAR-100 dataset, at 0.7133.

\subsection{Performance on other data flows.}
Table \ref{tab:ablation_training_test_setup} is divided into two main categories, including the first one is the first two rows that show the results of models trained locally, and the second one is the last three rows that illustrate collaboration results between these models. The purpose of this experiment is to approximate the lower bounds and upper bounds of the models' performance when using different training and testing data setups. Regarding the first group, the independent local training scenario without FL is conducted to form a strong baseline. For the second group, FL collaboration procedures with and without using private data as the clients' training sets are performed to demonstrate the necessity of local adaptation.

\subsection{Trade-off between Synthetic Data Privacy Levels and Model Performance}
We evaluate our proposed framework under different users' data privacy budget $\epsilon$, from the low range $\epsilon
\in [1 \ldots 5] $ to a higher range between $[6, \ldots, 10]$, and lastly a scenario where the synthetic data is generated by a generative model without having any added DP-noise. The quality of those aforementioned synthetic data sets is evaluated using Inception Score (IS) and Frechet Inception Distance (FID) \cite{heusel2017gans}. The results shown in Table \ref{tab:ablation_synthetic_quality} and Fig. \ref{fig:ablation_synthetic_chart} indicate that the higher DP-noise added in the synthetic data might result in higher generalization error.

\begin{table}[H]
\caption{Test accuracy and Macro F1-score with respected to synthetic data quality.}

\begin{tabular}{l|c|c|c|c}
                             & \textbf{BMTA} $\uparrow$ & \textbf{BMT-F1} $\uparrow$ & \textbf{IS} $\uparrow$ & \textbf{FID} $\downarrow$ \\ \hline
$\epsilon \in [1 \ldots 5]$  & 0.8133                 & 0.7454                & 1.2242          & 407.5517           \\
$\epsilon \in [6 \ldots 10]$ & 0.8400                 & 0.7940                & 1.8100          & 249.0475           \\
$\epsilon = \infty$          & 0.8533                 & 0.8199                & 2.9442          & 49.9617            \\ \hline
\end{tabular}

\label{tab:ablation_synthetic_quality}
\end{table}

\section{Conclusions and Future Works}
\label{sec:discussion}

In this work, we show that our proposed Personalized Privacy-Preserving Federated Learning (PPPFL) framework could not only minimize the risk of sensitive information being leaked but also overcome the challenges of the nonindependent and identically distributed (non-IID) data among clients. We demonstrate the efficiency of the PPPFL framework as it outperforms many existing baselines on both DP-synthetic and non-private synthetic data when evaluating different datasets, including MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. We also indicate the existence of a trade-off between synthetic data privacy levels and the model's performance. Despite our framework's performance improvements, conducting experiments with other GANs architectures is necessary.

\textbf{User incentive to participate in the protocol.} In this part of our work, we focus on the incentives for users to participate in the PPPFL framework. In cross-silo FL, organizations or companies usually have intentionally long-term strategic focuses and development goals. This makes the long-term cooperation relationship more possible, as long as we can design a proper incentive mechanism. To this end, we demonstrate a strong correlation between the local performance of a user's model and the benefits that each client derives from participating in the protocol. This correlation is shown in Fig. \ref{fig:incentive_changes_correlation}. Particularly, if local models have lower performance, it is evident that they would gain more from participating in the PPPFL protocol.

This phenomenon raises the possibility of fairness issues within the PPPFL framework, which we plan to address in future work.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi

This work was supported by VinUni-Illinois Smart Health Center (VISHC), VinUniversity.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

redpajama_set_name: RedPajamaArXiv