summaries: The paper focuses on the topic of differentially private deep learning. Specifically, based on the deep residual learning, they first see it as an ODE. Then, to reduce the reversibility of the ODE, they modify the model as an SDE. By discretizing the SDE, they get a perturbed version of residual learning and use this to design DP-algorithms. The first strategy is directly followed the SDE while the second strategy is with an addition multiplicative noise of the additive noise. Finally, they show that their methods to defend membership inference attack both theoretically and practically.  The paper presents a method for training ResNets with differential privacy. Rather than the usual methods based on noisy gradient descent, the authors propose adding noise at each layer of the network during both training and testing. The authors prove differential privacy guarantees for two strategies of this type (one with additive and one with multiplicative noise). They also show some evidence that the noise can help generalization, by showing that the Rademacher complexity of a continuous linearized version of the model is lower when noise is added. This paper studies an important problem and proposes the novel residual perturbation to protect privacy while maintaining the ResNet models’ utility.  Two SDE models are provided to inject noises with abundant theoretical proof are provided. Experimental results demonstrate the performance of privacy protection and classification accuracy on benchmark datasets. My major concern is about the utility enhancement and the DP guarantee (see cons below). Hope the authors can address my concern in the rebuttal period.
abstractText: Protecting data privacy in deep learning (DL) is at its urgency. Several celebrated privacy notions have been established and used for privacy-preserving DL. However, many of the existing mechanisms achieve data privacy at the cost of significant utility degradation. In this paper, we propose a stochastic differential equation principled residual perturbation for privacy-preserving DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap for DL. Empirically, we show that residual perturbation outperforms the stateof-the-art DP stochastic gradient descent (DPSGD) in both membership privacy protection and maintaining the DL models’ utility. For instance, in the process of training ResNet8 for the IDC dataset classification, residual perturbation obtains an accuracy of 85.7% and protects the perfect membership privacy; in contrast, DPSGD achieves an accuracy of 82.8% and protects worse membership privacy.
references.authors: Martin Abadi Andy Chu Ian Goodfellow H. McMahan Ilya Mironov Kunal Talwar Li Zhang
references.title: Deep Learning with Differential Privacy
references.venue: In 23rd ACM Conference on Computer and Communications Security (CCS 2016),
references.authors: Martin Abadi Andy Chu Ian Goodfellow H Brendan McMahan Ilya Mironov Kunal Talwar Li Zhang
references.title: Deep learning with differential privacy
references.venue: In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security,
references.authors: Mohammad Al-Rubaie J Morris Chang
references.title: Reconstruction attacks against mobile-based continuous authentication systems in the cloud
references.venue: IEEE Transactions on Information Forensics and Security,
references.authors: Ho Bae Dahuin Jung Sungroh Yoon
references.title: Anomigan: Generative adversarial networks for anonymizing private medical data
references.venue: arXiv preprint arXiv:1901.11313,
references.authors: Eugene Bagdasaryan Omid Poursaeed Vitaly Shmatikov
references.title: Differential privacy has disparate impact on model accuracy
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Peter L Barlett Sahar Mendelson
references.title: Rademacher and gaussian complexities: Risk bounds and structural results
references.venue: Jounal of Machine Learning Research,
references.authors: Marco Barreno Blaine Nelson Russell Sears Anthony D Joseph J Doug Tygar
references.title: Can machine learning be secure
references.venue: In Proceedings of the 2006 ACM Symposium on Information, computer and communications security,
references.authors: Raef Bassily Adam Smith Abhradeep Thakurta
references.title: Private empirical risk minimization: Efficient algorithms and tight error bounds
references.venue: In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science,
references.authors: Kamalika Chaudhuri Claire Monteleoni Anand D Sarwate
references.title: Differentially private empirical risk minimization
references.venue: Journal of Machine Learning Research,
references.authors: Jinshuo Dong Aaron Roth Weijie J Su
references.title: Gaussian differential privacy
references.venue: arXiv preprint arXiv:1905.02383,
references.authors: Cynthia Dwork Frank McSherry Kobbi Nissim Adam Smith
references.title: Calibrating noise to sensitivity in private data analysis
references.venue: In Theory of cryptography conference,
references.authors: Khaled El Emam Fida Kamal Dankar
references.title: Protecting privacy using k-anonymity
references.venue: Journal of the American Medical Informatics Association,
references.authors: Wei Feng Zheng Yan Hengrun Zhang Kai Zeng Yu Xiao Y. Hou
references.title: A Survey on Security, Privacy and Trust in Mobile Crowdsourcing
references.venue: IEEE Internet of Things Journal,
references.authors: Matt Fredrikson Somesh Jha Thomas Ristenpart
references.title: Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
references.venue: In 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS 2015),
references.authors: Matt Fredrikson Somesh Jha Thomas Ristenpart
references.title: Model inversion attacks that exploit confidence information and basic countermeasures
references.venue: In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security,
references.authors: Matthew Fredrikson Eric Lantz Somesh Jha Simon Lin David Page Thomas Ristenpart
references.title: Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing
references.venue: In 23rd {USENIX} Security Symposium ({USENIX} Security
references.authors: Neil Zhenqiang Gong Bin Liu
references.title: You are who you know and how you behave: Attribute inference attacks via users’ social friends and behaviors
references.venue: In 25th {USENIX} Security Symposium ({USENIX} Security
references.authors: Neil Zhenqiang Gong Bin Liu
references.title: Attribute inference attacks in online social networks
references.venue: ACM Transactions on Privacy and Security (TOPS),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Ehsan Hesamifard Hassan Takabi Mehdi Ghasemi Rebecca N Wright
references.title: Privacy-preserving machine learning as a service
references.venue: Proceedings on Privacy Enhancing Technologies,
references.authors: Desmond J Higham
references.title: An algorithmic introduction to numerical simulation of stochastic differential equations
references.venue: SIAM review,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Fima C Klebaner
references.title: Introduction to stochastic calculus with applications
references.authors: Jakub Konečnỳ H Brendan McMahan Felix X Yu Peter Richtárik Ananda Theertha Suresh Dave Bacon
references.title: Federated learning: Strategies for improving communication efficiency
references.venue: arXiv preprint arXiv:1610.05492,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.authors: Bai Li Changyou Chen Hao Liu Lawrence Carin
references.title: On connecting stochastic gradient mcmc and differential privacy
references.venue: In The 22nd International Conference on Artificial Intelligence and Statistics,
references.authors: Zhicong Liang Bao Wang Quanquan Gu Stanely Osher Yuan Yao
references.title: Exploring private federated learning with laplacian smoothing
references.venue: arXiv preprint arXiv:2005.00218,
references.authors: Yehuda Lindell Benny Pinkas
references.title: Privacy preserving data mining
references.venue: In Annual International Cryptology Conference,
references.authors: Xuanqing Liu Tesi Xiao SiSi Qin Cao Sanjiv Kumar Cho-Jui Hsieh
references.title: Neural sde: Stabilizing neural ode networks with stochastic noise
references.authors: Yun Liu Krishna Gadepalli Mohammad Norouzi George E. Dahl Martin C. Stumpe
references.title: Detecting Cancer Metastases on Gigapixel
references.venue: Pathology Images
references.authors: Ledoux M Talagrand
references.title: Probability in Banach spaces: Isoperimetry and processes
references.authors: H Brendan McMahan Eider Moore Daniel Ramage Seth Hampson
references.title: Communication-efficient learning of deep networks from decentralized data
references.venue: arXiv preprint arXiv:1602.05629,
references.authors: H Brendan McMahan Galen Andrew Ulfar Erlingsson Steve Chien Ilya Mironov Nicolas Papernot Peter Kairouz
references.title: A general approach to adding differential privacy to iterative training procedures
references.venue: arXiv preprint arXiv:1812.06210,
references.authors: Ilya Mironov
references.title: Rényi differential privacy
references.venue: IEEE 30th Computer Security Foundations Symposium (CSF),
references.authors: Nicolas Papernot Martı́n Abadi Lfar Erlingsson Ian Goodfellow Kunal Talwar
references.title: Semisupervised Knowledge Transfer for Deep Learning from Private Training Data
references.venue: In 5th International Conference on Learning Representation (ICLR
references.authors: Nicolas Papernot Shuang Song Ilya Mironov Ananth Raghunathan Lfar Erlingsson
references.title: Scalable Private Learning with PATE
references.venue: In International Conference on Learning Representations (ICLR
references.authors: Adnan Siraj Rakin Zhezhi He Deliang Fan
references.title: Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack
references.venue: arXiv preprint arXiv:1811.09310,
references.authors: Alexandre Sablayrolles Matthijs Douze Cordelia Schmid Hervé Jégou
references.title: D\’ej\a vu: an empirical evaluation of the memorization properties of convnets
references.venue: arXiv preprint arXiv:1809.06396,
references.authors: Ahmed Salem Yang Zhang Mathias Humbert Pascal Berrang Mario Fritz Michael Backes
references.title: Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models
references.venue: arXiv preprint arXiv:1806.01246,
references.authors: Reza Shokri Vitaly Shmatikov
references.title: Privacy-Preserving Deep Learning
references.venue: In 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS 2015),
references.authors: Reza Shokri Marco Stronati Congzheng Song Vitaly Shmatikov
references.title: Membership inference attacks against machine learning models
references.venue: IEEE Symposium on Security and Privacy (SP),
references.authors: Latanya Sweeney
references.title: k-anonymity: A model for protecting privacy
references.venue: International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems,
references.authors: Florian Tramèr Fan Zhang Ari Juels Michael K Reiter Thomas Ristenpart
references.title: Stealing machine learning models via prediction apis
references.venue: In 25th {USENIX} Security Symposium ({USENIX} Security
references.authors: Bao Wang Quanquan Gu March Boedihardjo Farzin Barekat Stanley J Osher
references.title: Dp-lssgd: A stochastic optimization method to lift the utility in privacy-preserving erm
references.venue: arXiv preprint arXiv:1906.12056,
references.authors: Bao Wang Zuoqiang Shi Stanley Osher
references.title: Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Yu-Xiang Wang Stephen Fienberg Alex Smola
references.title: Privacy for free: Posterior sampling and stochastic gradient monte carlo
references.venue: In International Conference on Machine Learning,
references.authors: Zhibo Wang Mengkai Song Zhifei Zhang Yang Song Qian Wang Hairong Qi
references.title: Beyond inferring class representatives: User-level privacy leakage from federated learning
references.venue: In IEEE INFOCOM 2019-IEEE Conference on Computer Communications,
references.authors: Bingzhe Wu Chaochao Chen Shiwan Zhao Cen Chen Yuan Yao Guangyu Sun Li Wang Xiaolu Zhang Jun Zhou
references.title: Characterizing membership privacy in stochastic gradient langevin dynamics
references.venue: arXiv preprint arXiv:1910.02249,
references.authors: Bingzhe Wu Shiwan Zhao Guangyu Sun Xiaolu Zhang Zhong Su Caihong Zeng Zhihong Liu
references.title: P3sgd: Patient privacy preserving sgd for regularizing deep cnns in pathological image classification
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Samuel Yeom Irene Giacomelli Matt Fredrikson Somesh Jha
references.title: Privacy risk in machine learning: Analyzing the connection to overfitting
references.venue: IEEE 31st Computer Security Foundations Symposium (CSF),
references.authors: Yuen Man-Ching King Irwin Leung Kwong-Sak
references.title: A Survey of Crowdsourcing Systems
references.venue: In Proceedings of the IEEE international conference on social computing (Socialcom
references.authors: Xu Zheng Zhipeng Cai Yingshu Li
references.title: Data linkage in smart internet of things systems: A consideration from a privacy perspective
references.venue: IEEE Communications Magazine,
references.authors: Ligeng Zhu Zhijian Liu Song Han
references.title: Deep leakage from gradients
references.venue: In Advances in Neural Information Processing Systems,
sections.heading: 1 INTRODUCTION
sections.text: Many high-capacity deep nets (DNs) are trained with private data, including medical images and financial transaction data (Yuen et al., 2011; Feng et al., 2017; Liu et al., 2017). DNs usually overfit and can memorize the private training data, which makes training DNs exposed to data privacy leakage (Fredrikson et al., 2015a; Shokri et al., 2017; Salem et al., 2018; Yeom et al., 2018; Sablayrolles et al., 2018). Given a pre-trained DN, the membership inference attack can determine if an instance is in the training set based on DN’s response (Fredrikson et al., 2014; Shokri et al., 2017; Salem et al., 2018); the model extraction attack can learn a surrogate model that matches the target model, given the adversary only black-box access to the target model (Tramèr et al., 2016; Gong & Liu, 2018); the model inversion attack can infer certain features of a given input from the output of a target model (Fredrikson et al., 2015b; Al-Rubaie & Chang, 2016); the attribute inference attack can deanonymize the anonymized training data (Gong & Liu, 2016; Zheng et al., 2018).
  Machine learning (ML) with data privacy is crucial in many applications (Lindell & Pinkas, 2000; Barreno et al., 2006; Hesamifard et al., 2018; Bae et al., 2019). Several algorithms have been developed to reduce privacy leakage include differential privacy (DP) (Dwork et al., 2006), federated learning (FL) (McMahan et al., 2016; Konečnỳ et al., 2016), and k-anonymity (Sweeney, 2002; El Emam & Dankar, 2008). Objective, output, and gradient perturbations are among the most used approaches for ML with DP guarantees at the cost of significant utility degradation (Chaudhuri et al., 2011; Bassily et al., 2014; Shokri & Shmatikov, 2015; Abadi et al., 2016b; Bagdasaryan et al., 2019). FL trains centralized ML models, through gradient exchange, with the training data being distributed at the edge devices. However, the gradient exchange can still leak the privacy (Zhu et al., 2019; Wang et al., 2019c). Most of the existing privacy is achieved at a tremendous sacrifice of utility. Moreover, training ML models using the state-of-the-art DP stochastic gradient descent (DPSGD) leads to tremendous computational cost due to the requirement of computing and clipping the per-sample gradient (Abadi et al., 2016a). It remains a great interest to develop new privacy-preserving ML algorithms without excessive computational overhead or degrading the utility of the ML models.
sections.heading: 1.1 OUR CONTRIBUTION
sections.text: In this paper, we propose residual perturbation for privacy-preserving deep learning (DL) with DP guarantees. At the core of residual perturbation is injecting Gaussian noise to each residual mapping
  of ResNet (He et al., 2016), and the residual perturbation is theoretically principled by the stochastic differential equation (SDE) theory. The major advantages of residual perturbation are threefold: • It can protect the membership privacy of the training data almost perfectly and often without
  sacrificing ResNets’ utility. Furthermore, it can even improve ResNets’ classification accuracy. • It has fewer hyperparameters to tune than the benchmark DPSGD. Also, it is more computationally
  efficient than DPSGD, which requires to compute the per-sample gradient. • It can be easily implemented by a few lines of code in modern DL libraries.
sections.heading: 1.2 RELATED WORK
sections.text: Improving the utility of ML models with DP guarantees is an important task. PATE (Papernot et al., 2017; 2018) uses semi-supervised learning together with model transfer between the “student” and “teacher” models to enhance utility. Several variants of the DP notions have also been proposed to improve the privacy budget and some times can also improve the resulting model’s utility at a given DP budget (Abadi et al., 2016b; Mironov, 2017; Wang et al., 2018; Dong et al., 2019). Some post-processing techniques have also been developed to improve the utility of ML models with negligible computational overhead (Wang et al., 2019a; Liang et al., 2020). From the SDE viewpoint, (Li et al., 2019; Wang et al., 2015) showed that several stochastic gradient Monte Carlo samplers could reach state-of-the-art performance in terms of both privacy and utility in Bayesian learning.
  Gaussian noise injection in residual learning has been used to improve the robustness of ResNets (Rakin et al., 2018; Wang et al., 2019b; Liu et al., 2019). In this paper, we inject Gaussian noise to each residual mapping to achieve data privacy instead of adversarial robustness.
sections.heading: 1.3 ORGANIZATION
sections.text: We organize this paper as follows: In Section 2, we introduce the residual perturbation for privacypreserving DL. In Section 3, we present the generalization and DP guarantees for residual perturbation. In Section 4, we numerically verify the efficiency of the residual perturbation in protecting data privacy without degrading the underlying models’ utility. We end with some concluding remarks. Technical proofs and some more experimental details and results are provided in the appendix.
sections.heading: 1.4 NOTATIONS
sections.text: We denote scalars by lower or upper case letters; vectors/ matrices by lower/upper case bold face letters. For a vector x = (x1, · · · , xd) ∈ Rd, we use ‖x‖2 = ( ∑d i=1 |xi|2)1/2 to denote its `2 norm. For a matrix A, we use ‖A‖2 to denote its induced norm by the vector `2 norm. We denote the standard Gaussian in Rd as N (0, I) with I ∈ Rd×d being the identity matrix. The set of (positive) real numbers is denoted as (R+) R. We use B(0, R) to denote the ball centered at 0 with radius R.
sections.heading: 2 ALGORITHMS
sections.heading: 2.1 DEEP RESIDUAL LEARNING AND ITS CONTINUOUS ANALOGUE
sections.text: Given the training set SN := {xi, yi}Ni=1, with {xi, yi} ⊂ Rd × R being a data-label pair. For a given xi the forward propagation of a ResNet with M residual mappings can be written as
  xl+1 = xl + F̂ (xl,Wl), for l = 0, 1, · · · ,M − 1, with x0 = xi; ŷi = f(xM ), (1)
  where F̂ (·,Wl) is the nonlinear mapping of the lth residual mapping parameterized by Wl; f is the output activation function, and ŷi is the predicted label for xi. The heuristic continuum limit of (1) is
  dx(t) = F (x(t),W(t))dt, x(0) = x̂, where t is the time variable. (2)
  The ordinary differential equation (ODE) (2) can be revertible, and thus the ResNet counterpart might be exposed to data privacy leakage. For instance, we use the ICLR logo (Fig. 1 (a)) as the initial data x̂ in (2). Then we simulate the forward propagation of ResNet by solving (2) from t = 0 to t = 1 using the forward Euler solver with a time step size ∆t = 0.01 and a given velocity field F (x(t),W(t)) (see Appendix E for the details of F (x(t),W(t))), which maps the original image to its features (Fig. 1 (b)). To recover the original image, we start from the feature and use the backward Euler iteration, i.e., x̃(t) = x̃(t+∆t)−∆tF (x̃(t+∆t), t+∆t), to evolve x̃(t) from t = 1 to t = 0 with x̃(1) = x(1) being the features obtained in the forward propagation. We plot the recovered image from features in Fig. 1 (c), and the original image can be almost perfectly recovered.
sections.heading: 2.2 RESIDUAL PERTURBATION AND ITS SDE ANALOGUE
sections.text: In this part, we propose two SDE models to reduce the reversibility of (2), and the corresponding residual perturbations analogue can protect the data privacy in DL.
  Strategy I. For the first strategy, we consider the following SDE model:
  dx(t) = F (x(t),W(t))dt+ γdB(t), γ > 0, (3)
  where B(t) is the standard Brownian motion. We simulate the forward propagation and reverseengineering the input from the output by solving the SDE model (3) with γ = 1 using the same F (x(t),W(t)) and initial data x̂. We use the following forward (4) and backward (5) EulerMaruyama discretizations (Higham, 2001) of (3),
  x(t+ ∆t) = x(t) + ∆tF (x(t),W(t)) + γN (0, √ ∆t I), (4)
  x̃(t) = x̃(t+ ∆t)−∆tF (x̃(t+ ∆t),W(t+ ∆t)) + γN (0, √ ∆t I), (5)
  for the forward and backward propagation, respectively. Figure 1 (d) and (e) show the results of the forward and backward propagation by SDE, respectively, and these results show that it is much harder to reverse the features obtained by SDE evolution. The SDE model informs us to inject Gaussian noise, in both training and test phases, to each residual mapping of ResNet to protect data privacy, which results in
  xi+1 = xi + F̂ (xi,Wi) + γni, where ni ∼ N (0, I)1. (6)
  Strategy II. For the second strategy, we consider using the multiplicative noise instead of the additive noise used in (3) 2 and (6), and the corresponding SDE can be written as
  dx(t) = F (x(t),W(t))dt+ γx(t) dB(t), γ > 0, (7)
  where denotes the Hadamard product. Similarly, we can use the forward and backward EulerMaruyama discretizations of (7) to propagate the image in Fig. 1 (a), and we provide these results in Appendix D.1. The corresponding residual perturbation is
  xi+1 = xi + F̂ (xi,Wi) + γxi ni, where ni ∼ N (0, I), (8)
  again, the noise γxi ni is injected to each residual mapping in both training and test phases. We will provide theoretical guarantees for these two residual perturbation schemes, i.e., (6) and (8), in Section 3, and numerically verify their efficacy in Section 4.
sections.heading: 2.3 UTILITY ENHANCEMENT VIA MODEL ENSEMBLE
sections.text: Wang et al. (2019b) showed that an ensemble of noise injected ResNets can improve models’ utility. In this paper, we will also study the model ensemble for utility enhancement. We inherit notations from (Wang et al., 2019b), e.g., we denote an ensemble of two noise injected ResNet8 as En2ResNet8.
sections.heading: 3 MAIN THEORY
sections.text: In this section, we will provide theoretical guarantees for the above two residual perturbations. 1Liu et al. (2019); Wang et al. (2019b) used this noise injection to improve robustness of ResNets. 2Liu et al. (2019) injected multiplicative noise to neural networks to improve their robustness.
sections.heading: 3.1 DIFFERENTIAL PRIVACY GUARANTEE FOR STRATEGY I
sections.text: We consider the following function class for ResNets with residual perturbation:
  F1 := {f(x) = wTxM |xi+1 = xi + φ ( Uixi ) + γni, i = 0, · · · ,M − 1, (9)
  x0 = input data + πn,n and ni ∼ N (0, I),w ∈ Rd,Ui ∈ Rd×d},
  where x0 ∈ Rd is the noisy input 3, Ui is the weight matrix in the ith residual mapping and w ∈ Rd is the weights of the last layer. γ, π > 0 are hyperparameters. φ = BN(ψ) with BN being the batch normalization and ψ being a L-Lipschitz and monotonically increasing activation function (e.g., ReLU). We first recap on the definition of differential privacy below. Definition 1 (( , δ)-DP). (Dwork et al., 2006)A randomized mechanism M : SN → R satisfies ( , δ)-DP if for any two datasets S, S′ ∈ SN that differ by one element, and any output subset O ⊆ R, it holds that P[M(S) ∈ O] ≤ e · P[M(S′) ∈ O] + δ,where δ ∈ (0, 1) and > 0.
  We have the following DP guarantee for Strategy I, and we provide its proof in Appendix A. Theorem 1. Assume the input to ResNet lies in B(0, R) and the output of every residual mapping is normal distributed and bounded by G, in `2 norm, in expectation. Given the total number of iterations T used for training ResNet. For any > 0 and δ, λ ∈ (0, 1), the parameters Ui and w in the ResNet with residual perturbation satisfies ((λ/i+ (1− λ)) , δ)-DP and ((λ/M + (1− λ)) , δ)DP, respectively, provided that π > R √ (2Tbα)/(Nλ ) and γ > G √ (2Tbα)/(Nλ ), where α = log(1/δ)/ ((1− λ) ) + 1, M,N and b are the number of residual mappings, training data, and batch size, respectively. In particular, when γ > G √ (2Tbα)/(NMλ ) the whole model obtained by injecting noise according to strategy I satisfies ( , δ)-DP.
sections.heading: 3.2 THEORETICAL GUARANTEES FOR STRATEGY II
sections.text: Privacy. To analyze the residual perturbation (8), we consider the following function class:
  F2 := {f (x) = wTxM + πxMn| xi+1 = xi + φ ( Uixi ) + γx̃i ni), (10)
  i = 0, · · · ,M − 1,ni ∼ N (0, I), ‖w‖2 ≤ a}
  where a > 0 is a constant; we denote the entry of xi that has the largest absolute value as ximax, and x̃i is defined as (sgn(xij) max(|xij |, η))dj=1. Due to batch normalization, we assume φ can be bounded by a positive constant B. The other notations are defined similar to that in (9).
  Consider training F2 by using two different datasets S and S′, and we denote the resulting models as: f (x|S) := wT1 xM + πxMnM ; xi+1 = xi + φ ( Ui1x i )
  + γx̃i ni, i = 0, · · · ,M − 1. (11) f ( x|S ′ )
  := wT2 x M + πxMnM ; xi+1 = xi + φ ( Ui2x i ) + γx̃i ni, i = 0, · · · ,M − 1. (12)
  Theorem 2. For f(x|S) and f(x|S′) that are defined in (11) and (12), respectively. Let λ ∈ (0, 1), δ ∈ (0, 1), and > 0, if γ > (B/η) √ (2αM)/(λ ) and π > a √ (2αM) /λ , where α = log(1/δ)/ ((1− λ) ) + 1, then P[f(x|S) ∈ O] ≤ e · P[f(x|S′ ] ∈ O] + δ for any input x and any subset O in the output space.
  We provide the proof of Theorem 2 in Appendix B. Theorem 2 guarantees the privacy of the training data given only black-box access to the model, i.e., the model will output the prediction for any input without granting adversaries access to the model itself. In particular, we cannot infer whether the model is trained on S or S′ no matter how we query the model in a black-box fashion. We leave the theoretical DP-guarantee for for Strategy II as a future work.
  Generalization Gap. Many works have shown that overfitting in training ML models leads to privacy leakage (Salem et al., 2018), and reducing overfitting can mitigate data privacy leakage (Shokri et al., 2017; Yeom et al., 2018; Sablayrolles et al., 2018; Salem et al., 2018; Wu et al., 2019b). In this part, we will show that the residual perturbation (8) can reduce overfitting via computing the Rademacher complexity. For simplicity, we consider binary classification problems. Suppose SN ={xi, yi}Ni=1 is drawn from X × Y ⊂ Rd×{−1,+1} with X and Y being the input data and label
  3We add noise to the input for DP guarantee
  spaces, respectively. Assume D is the underlying distribution of X × Y , which is unknown. Let H ⊂ V be the hypothesis class of the ML model. We first recap on the definition of Rademacher complexity. Definition 2. (Barlett & Mendelson, 2002) Let H : X → R be the space of real-valued functions on the space X . For a given sample S = {x1,x2, · · · ,xN} of size N , the empirical Rademacher complexity ofH is defined as
  RS(H) := 1
  N Eσ[sup
  h∈H N∑ i=1 σih(xi)],
  where σ1, σ2, · · · , σN are i.i.d. Rademacher random variables with P(σi = 1) = P(σi = −1) = 12 .
  Rademacher complexity is a tool to bound the generalization gap (Barlett & Mendelson, 2002). The smaller the generalization gap is, the less overfitting the model is. For ∀xi ∈ Rd and constant c ≥ 0, we consider the following two function classes:
  F := {f(x,w) = wxp(T )|dx(t) = Ux(t)dt,x(0) = xi; w ∈ R1×d,U ∈ Rd×d} with ‖w‖2, ‖U‖2 ≤ c},
  G := {f(x,w) = E (wxp(T )) |dx(t) = Ux(t)dt+ γx(t) dB(t),x(0) = xi; w ∈ R1×d, with U ∈ Rd×d, ‖w‖2, ‖U‖2 ≤ c},
  where 0 < p < 1 takes the value such that xp is well defined on the whole Rd. γ > 0 is a hyperparameter and U is a circulant matrix that corresponding to the convolution layer in DNs. B(t) being the 1D Brownian motion. The function class F represents the continuous analogue of ResNet without inner nonlinear activation functions, and G denotes F with the residual perturbation (8). Theorem 3. Given the training set SN = {xi, yi}Ni=1. We have RSN (G) < RSN (F).
  We provide the proof of Theorem 3 in Appendix C, where we will also provide quantitative lower and upper bounds of the above Rademacher complexities. Theorem 3 shows that residual perturbation (8) can reduce the generalization error. We will numerically verify this generalization error reduction for ResNet with residual perturbation in Section 4.
sections.heading: 4 EXPERIMENTS
sections.text: In this section, we will numerically verify that 1) can residual perturbation protect data privacy; in particular, membership privacy? 2) can the ensemble of ResNets with residual perturbation improve the classification accuracy? 3) are skip connections crucial in residual perturbation for DL with data privacy? 4) what is the advantage of the residual perturbation over the DPSGD? We focus on Strategy I in this section, and we provide the results of Strategy II in Appendix D.
sections.heading: 4.1 PRELIMINARIES
sections.text: Datasets. We consider both CIFAR10/CIFAR100 (Krizhevsky et al., 2009) and the Invasive Ductal Carcinoma (IDC) datasets. Both CIFAR10 and CIFAR100 contain 60K 32× 32 color images with 50K and 10K of them used for training and test, respectively. The IDC dataset is a breast cancerrelated benchmark dataset, which contains 277,524 patches of the size 50× 50 with 198,738 labeled negative (0) and 78,786 labeled positive (1). Figure 2 depicts a few patches from the IDC dataset. For the IDC dataset, we follow Wu et al. (2019a) and split the whole dataset into training, validation, and test set. The training set consists of 10,788 positive patches and 29,164 negative patches, and the test set contains 11,595 positive patches and 31,825 negative patches. The remaining patches are used as the validation set. For each dataset, we split its training set into Dshadow and Dtarget with the same size. Furthermore, we split Dshadow into two halves with the same size and denote them as Dtrainshadow and Doutshadow, and split Dtarget by half into D train target and D out target. The purpose of this splitting of the training set is for the membership inference attack, which will be discussed below.
  Membership inference attack. To verify the efficiency of residual perturbation for protecting data privacy, we consider the membership inference attack (Salem et al., 2018) in all the experiments below. The membership attack proceeds as follows: 1) train the shadow model by using Dtrainshadow; 2) apply the trained shadow model to predict all data points in Dshadow and obtain the corresponding
  classification probabilities of belonging to each class. Then we take the top three classification probabilities (or two in the case of binary classification) to form the feature vector for each data point. A feature vector is tagged as 1 if the corresponding data point is in Dtrainshadow, and 0 otherwise. Then we train the attack model by leveraging all the labeled feature vectors; 3) train the target model by using Dtraintarget and obtain feature vector for each point in Dtarget. Finally, we leverage the attack model to decide whether a data point is in Dtraintarget.
  Experimental settings. We consider En5ResNet8 (ensemble of 5 ResNet8 with residual perturbation) and the standard ResNet8 as the target and shadow models. We use a multilayer perceptron with a hidden layer of 64 nodes, followed by a softmax output function as the attack model, which is adapted from (Salem et al., 2018). We apply the same settings as that used in (He et al., 2016) to train the target and shadow models on the CIFAR10 and CIFAR100. For training models on the IDC dataset, we run 100 epochs of SGD with the same setting as before except that we decay the learning rate by 4 at the 20th, 40th, 60th, and 80th epoch, respectively. Moreover, we run 50 epochs of Adam (Kingma & Ba, 2014) with a learning rate of 0.1 to train the attack model. For both IDC and CIFAR datasets, we set π as half of γ in Strategy I, which simplifies hyperparameters calibration, and based on our experiment it gives a good trade-off between privacy and accuracy.
  Performance evaluations. We consider both classification accuracy and capability for protecting membership privacy. The attack model is a binary classifier, which is to decide if a data point is in the training set of the target model. For any x ∈ Dtarget, we apply the attack model to predict its probability (p) of belonging to the training set of the target model. Given any fixed threshold t if p ≥ t, we classify x as a member of the training set (positive sample), and if p < t, we conclude that x is not in the training set (negative sample); so we can obtain different attack results with different thresholds. Furthermore, we can plot the ROC curve(see details in subsection 4.5) of the attack model and use the area under the ROC curve (AUC) as an evaluation of the membership inference attack. The target model protects perfect membership privacy if the AUC is 0.5 (attack model performs random guess), and the higher AUC is, the less private the target model is. Moreover, we use the precision (the fraction of records inferred as members are indeed members of the training set) and recall (the fraction of training set that is correctly inferred as members of the training set by the attack model) to measure ResNets’ capability for protecting membership privacy.
sections.heading: 4.2 EXPERIMENTS ON THE IDC DATASET
sections.text: In this subsection, we numerically verify that the residual perturbation in protecting data privacy while retaining the classification accuracy on the IDC dataset. We select the En5ResNet8 as a benchmark architecture, which has ResNet8 as its baseline architecture (the details of the neural architectures are provided in Appendix F). As shown in Figure 3, we set four different thresholds to obtain different attack results with three different noise coefficients (γ) when γ = 0 means the standard ResNet8 without residual perturbation. We also depict the ROC curve for this experiment in Figure 7 (c).
  En5ResNet is remarkably better in protecting the membership privacy, and as γ increases the model becomes more resistant to the membership attack. Also, as the noise coefficient increases, the gap between training and test accuracies becomes smaller, which resonates with Theorem 3. For instance, when γ = 0.2 the AUC for the attack model is 0.495 and 0.563, respectively, for En5ResNet8 and ResNet8; the classification accuracy of En5ResNet8 and ResNet8 are 0.867 and 0.847, respectively.
sections.heading: 4.2.1 RESIDUAL PERTURBATION VS. DPSGD
sections.text: In this part, we compare the residual perturbation with the benchmark Tensorflow DPSGD module (McMahan et al., 2018), and we calibrate the hyperparameters, including the initial learning rate (0.1) which decays by a factor of 4 after every 20 epochs, noise multiplier (1.1), clipping threshold (1.0), micro-batches (128), and epochs (100) 4 such that the resulting model gives the optimal trade-off between membership privacy and classification accuracy. DPSGD is significantly more expensive due to the requirement of computing the per-sample gradient. We compare the standard ResNet8 trained by DPSGD with En1ResNet8 and En5ResNet8 with residual perturbation (γ = 0.3). Table 1 lists the AUC of the attack model and training and test accuracies of the target model; we see that residual perturbation can improve accuracy and protect better membership privacy.
sections.heading: 4.3 EXPERIMENTS ON THE CIFAR10/CIFAR100 DATASETS
sections.text: We further test the residual perturbation for ResNet8 and En5ResNet8 on the CIFAR10/CIFAR100 dataset. Figure 4 plots the performance of En5ResNet8 on the CIFAR10 dataset under the above four different measures. Again, the ensemble of ResNets with residual perturbation is remarkably less vulnerable to the membership inference attack; for instance, the AUC of the attack model for ResNet8 and En5ResNet8 (γ = 0.75) is 0.757 and 0.573, respectively. Also, the classification accuracy of En5ResNet8 (67.5%) is higher than that of ResNet8 (66.9%) for CIFAR10 classification. Figure 5 depicts the results of En5ResNet8 for CIFAR100 classification. These results confirm again that residual perturbation can protect membership privacy and improve classification accuracy.
  4https://github.com/tensorflow/privacy/tree/master/tutorials
sections.heading: 4.3.1 EFFECTS OF THE NUMBER OF MODELS IN THE ENSEMBLE
sections.text: In this part, we consider the effects of the number of residual perturbed ResNets in the ensemble. Figure 6 illustrates the performance of EnResNet8 for CIFAR10 classification measured in the AUC, and training and test accuracy. These results show that tuning the noise coefficient and the number of models in the ensemble is crucial to optimize the trade-off between accuracy and privacy.
sections.heading: 4.4 ON THE IMPORTANCE OF SKIP CONNECTIONS
sections.text: Residual perturbations theoretically relies on the irreversibility of the SDEs (3) and (7), and this ansatz lies in the skip connections in the ResNet. We test both standard ResNet and the modified ResNet without skip connections. For CIFAR10 classification, under the same noise coefficient (γ = 0.75), the test accuracy is 0.675 for the En5ResNet8 (with skip connection); while the test accuracy is 0.653 for the En5ResNet8 (without skip connection). Skip connections makes EnResNet more resistant to noise injection which is indeed crucial for the success of residual perturbation for protecting data privacy.
sections.heading: 4.5 ROC CURVES FOR EXPERIMENTS ON DIFFERENT DATASETS
sections.text: The receiver operating characteristic (ROC) curve can be used to illustrate the classification ability of a binary classifier. ROC curve is obtained by plotting the true positive rate against the false positive rate at different thresholds. The true positive rate, also known as recall, is the fraction of the positive set (all the positive samples) that is correctly inferred as a positive sample by the binary classifier. The false positive rate can be calculated by 1-specificity, where specificity is the fraction of the negative set (all the negative samples) that is correctly inferred as a negative sample by the binary classifier. In our case, the attack model is a binary classifier. Data points in the training set of the target model are tagged as positive samples, and data points out of the training set of the target model are tagged as negative samples. Then we plot ROC curves for different datasets (as shown in Figure 7). These ROC curves show that if γ is sufficiently large, the attack model’s prediction will be nearly a random guess.
sections.heading: 4.6 REMARK ON THE PRIVACY BUDGET
sections.text: In the experiments above, we set the constantsG andR to 30 for Strategy I. For classifying IDC with ResNet8, the DP budget for Strategy I is ( = 1.1e5, δ = 1e− 5) and the DP-budget for DPSGD is ( = 15.79, δ = 1e − 5). For classifying CIFAR10 with ResNet8, the DP budget for Strategy I is ( = 3e5, δ = 1e − 5) and the DP-budget for DPSGD is ( = 22.33, δ = 1e − 5). Note that theorem 1 offers a quite loose DP budget compared to DPSGD. There are several difficulties we need to overcome to get tight DP bounds for Strategy I. Compared to DPSGD, it is significantly harder. In particular, 1) the loss function of the nose injected ResNets is highly nonlinear and very complex with respect to the weights, also the noise term appears in the loss function due to the noise injected in each residual mapping. These together make the tight estimate very difficult. 2) In our proof, we leveraged the framework of subsampled Rényi-DP (Wang et al., 2018) to find a feasible range of
  noise variance parameter, and then convert to DP to get the value of γ for a given DP budget. This procedure will significantly reduce the accuracy of the estimated γ. We leave the tight DP guarantee as future work. In particular, how to reduce the accuracy of estimating due to the conversion between Rényi-DP and DP.
sections.heading: 5 CONCLUDING REMARKS
sections.text: In this paper, we proposed residual perturbations, whose theoretical foundation lies in the theory of stochastic differential equations, to protect data privacy for deep learning. Theoretically, we prove that the residual perturbation can reduce the generalization gap with differential privacy guarantees. Numerically, we have shown that residual perturbations are effective for protecting membership privacy on some benchmark datasets. In particular, on the IDC benchmark, residual perturbations protect better membership privacy than state-of-the-art differentially private stochastic gradient descent and achieve remarkably better classification accuracy.
sections.heading: A Proof of Theorem 1 13
sections.text: A.1 Rényi Differential Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 A.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
sections.heading: B Proof of Theorem 2 15
sections.heading: C Proof of Theorem 3 15
sections.text: C.1 Some Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 C.2 The proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
sections.heading: D Experiments on Strategy II 18
sections.text: D.1 Forward and Backward Propagation Using (7) . . . . . . . . . . . . . . . . . . 18 D.2 Experiments on the IDC Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 18 D.3 Experiments on the CIFAR10/CIFAR100 Datasets . . . . . . . . . . . . . . . . 18 D.4 ROC Curves for the Experiments on Different Datasets . . . . . . . . . . . . . . 19
sections.heading: E More Experimental Details 19
sections.heading: F Architectures of the Used DNs 20
sections.heading: A PROOF OF THEOREM 1
sections.text: A.1 RÉNYI DIFFERENTIAL PRIVACY
  We will use the notion of Rényi differential privacy (RDP) to prove the differential privacy (DP) guarantees for the proposed residual perturbations. First, let’s review the definition and several results of the Rényi differential privacy (Mironov, 2017). Definition 3. (Mironov, 2017) (Rényi divergence) For any two probability distributions P and Q defined over the distribution D, the Rényi divergence of order α > 1 is
  Dα(P ||Q) = 1
  α− 1 logEx∼Q(P/Q)α.
  Definition 4. (Mironov, 2017) ((α, )-RDP) A randomized mechanismM : D → R is said to have -Rényi differential privacy of order α or (α, )-RDP for short, if for any adjacent S,S ′ ∈ D that differ by only one entry, it holds that
  Dα(M(S)||M(S ′ ) ≤ ε.
  Lemma 1. (Mironov, 2017) Let f : D → R1 be (α, 1)-RDP and g : R1 ×D → R be (α, )-RDP, then the mechanism defined as (X,Y ), where X ∼ f(D) and Y ∼ g(X,D), satisfies (α, 1 + 2)- RDP. Lemma 2. (Mironov, 2017) (From RDP to ( , δ)-DP) If f is an (α, )-RDP mechanism, then it also satisfies ( − (log δ)/(α− 1), δ)-differential privacy for any 0 < δ < 1.
  Lemma 3. (Mironov, 2017) (Post-processing lemma) LetM : D → R be a randomized algorithm that is (α, )-RDP, and let f : R → R′ be an arbitrary randomized mapping. Then f(M(·)) : D → R′ is (α, )-RDP
  A.2 PROOF OF THEOREM 1
  In this subsection, we will give a proof of Theorem 1, i.e., DP-guarantee for the Strategy I.
  Proof. We will prove Theorem 1 by mathematical induction. Consider two adjacent datasets S = {x1, · · · ,xN−1,xN},S ′ = {x1, · · · ,xN−1, x̂N} that differ by one entry. For the first residual mapping, it is easy to check that when γ ≥ R √ 2α/ p we have Dα(x1N ||x̂1N ) = α‖xN − x̂N‖2/(2γ2) < εp. For the remaining residual mappings, we denote the response of the ith residual mapping, for any two input data xN and x̂N , as xiN , x̂ i N , respectively. Based on our assumpution, we have
  xiN + φ(U ixiN ) ∼ N (µN,i, σ2N,i)
  x̂iN + φ(U ix̂iN ) ∼ N (µ̂N,i, σ2N,i),
  where µN,i and µ̂N,i are both bounded by the constant G. If Dα(xiN ||x̂iN ) ≤ p/i, according the post-processing lemma (Lemma 3), we have
  Dα(x i N + φ(U ixiN )||x̂iN + φ(Uix̂iN )) = α‖µN,i − µ
  ′ N,i‖2
  2σ2N,i ≤ p/i,
  so when γ > √ 2αG2/ p, we further have
  Dα(x i N + φ(U ixiN ) + γn||x̂iN + φ(Uix̂iN ) + γn) = α‖µN,i − µ
  ′ N,i‖2
  2(σ2N,i + γ 2)
  ≤ p i+ 1 ,
  which implies that Dα(xi+1N ||x̂ i+1 N ) < p i+1 . On the other hand, note that ∇Ui+1`|xi+1N = l ′ ( xi+1N + φ ( Ui+1xi+1N )) φ ′ (Ui+1xi+1N )(x i+1 N ) T
  ∇Ui+1`|x̂i+1N = l ′ ( x̂i+1N + φ ( Ui+1x̂i+1N )) φ ′ (Ui+1x̂i+1N )(x̂ i+1 N ) T .
  Leveraging the post-processing lemma (Lemma 3) again, we get
  Dα(∇Ui+1`|xi+1N ||∇Ui+1`|x̂i+1N ) < p i+ 1 .
  Let Bt be the index set with |Bt| = b, and we update Ui as following:
  Ui+1t+1|S = Uit − α 1
  b ∑ j∈Bt ∇Ui+1t `(x i+1 j ,U i+1 t )
  Ui+1t+1|S ′ = Uit − α 1
  b ∑ j∈Bt ∇Ui+1t `(x i+1 j ,U i+1 t ),
  where Ui+1t is the weights updated after the tth training iterations. When N /∈ Bt, it’s obviously that Dα(Ui+1t+1|S||U i+1 t+1|S ′) = 0; when N ∈ Bt, the equations which we use to update Ui can be rewritten as
  Ui+1t+1|S = Uit − α(1/b) ∑
  j∈Bt−{N}
  ∇Ui+1t `(x i+1 j ,U i+1 t )− α(1/b)∇Ui+1t `(x i+1 N ,U i+1 t )
  Ui+1t+1|S ′ = Uit − α(1/b) ∑
  j∈Bt−{N}
  ∇Ui+1t `(x i+1 j ,U i+1 t )− α(1/b)∇Ui+1t `(x̂ i+1 N ,U i+1 t ).
  According to the post-processing lemma (Lemma 3), we have Dα(Ui+1t+1|S||U i+1 t+1|S ′) ≤ p/(i+ 1). Because there are only (Tb)/N steps where we use the information of xN and x̂N . Replace p by (Tb p)/N and use composition theorem we can get after T steps the output Ui+1T satisfies (α, p/(i + 1))-RDP and w satisfies (α, p/M)-RDP. By Lemma 2, we can easily establish the DP-guarantee for Strategy I, as stated in Theorem 1.
sections.heading: B PROOF OF THEOREM 2
sections.text: In this section, we will provide a proof for Theorem 2.
  Proof. Let φ = BN(ψ), where BN is batch normalization operation and ψ is an activation function. Because of the property of batch normalization, we assume that φ can be bounded by a positive constant B. To show that the model (9) guarantees training data privacy given only black-box access to the model. Consider training the model (9) with two different datasets S and S′, and we denote the resulting model as f(·|S) and f(·|S′), respectively. In the following, we prove that with appropriate choices of γ and π, Dα(f(x|S)||f(x|S ′ )) < p for any input x.
  First, we consider the convolution layers, and let Conv(x)i be the ith entry of the vectorized Conv(x). Then we have Conv(x)i = xi+φ(Ux)i+γx̃ini. For any two different training datasets, we denote
  Conv(x)i|S = xi + φ(U1x)i + γx̃ini ∼ N (xi + φ(U1x)i, γ2x̃2i ),
  and Conv(x)i|S ′ = xi + φ(U2x)i + γx̃ini ∼ N (xi + φ(U2x)i, γ2x̃2i ).
  Therefore, if γ > (B/η) √ (2αM)/( p), we have
  Dα ( N ( xi + φ (U1x)i , γ 2x̃2i ) ||N ( xi + φ (U2x)i , γ 2x̃2i ))
  ≤ α(φ(U1x)i − φ(U2x)i) 2 2γ2η2 ≤ 4αL 2B2‖x‖22 2γ2x̃2i ≤ 2αB 2
  γ2η2 ≤ p/M.
  Furthermore, γ > (b/η) √
  (2αM)/( p) guarantees (α, p/M)-RDP for every convolution layer. For the last fully connected layer, if π > a √ (2αM) / p, we have
  Dα ( N ( wT1 x, π 2x̃2i ) ||N ( wT2 x, π 2x̃2i ))
  = α ( wT1 x−wT2 x )2 2π2‖x‖2 ≤ 4αa 2‖x‖22 2π2‖x‖2
  ≤ 2αa 2
  π2 ≤ p/M,
  i.e., π > a √
  (2αM) / p guarantees that the fully connected layer to be (α, p/M)-RDP. According to Lemma 1, we have (α, p)-RDP guarantee for the ResNet of M residual mappings if γ > (Lb/η) √ (2αdM)/( p) and π > a √ (2αM) / p. Let λ ∈ (0, 1), for any given ( p, δ) pair, if
  p ≤ λ and −(log δ)/(α − 1), δ) ≤ (1 − λ) , then we have get the ( , δ)-DP guarantee for the ResNet with M residual mapping using the residual perturbation Strategy II.
sections.heading: C PROOF OF THEOREM 3
sections.text: In this section, we will proof that the residual perturbation (8) can reduce the generalization error via computing the Rademacher complexity. Let us first recap on some related lemmas on the stochastic differential equation and the Rademacher complexity.
  C.1 SOME LEMMAS
  Let ` : V × Y → [0, B] be the loss function. Here we assume ` is bounded and B is a positive constant. In addition, we denote the function class `H = {(x, y) → `(h(x), y) : h ∈ H, (x, y) ∈ X × Y }. The goal of the learning problem is to find h ∈ H such that the population risk R(h) = E(x,y)∈D[`(h(x), y)] is minimized. The gap between population risk and empirical risk RSN (h) = (1/N) ∑N i=1 `(h(xi), yi) is known as the generalization error. We have the following lemma and theorem to connect the population and empirical risks via Rademacher complexity.
  Lemma 4. (Ledoux-Talagrand inequality) (M & Talagrand, 2002)LetH be a bounded real valued function space and let φ : R→ R be a Lipschitz with constant L and φ(0) = 0. Then we have
  1 n Eσ [ sup h∈H n∑ i=1 σiφ (h (xi)) ] ≤ L n Eσ [ sup h∈H n∑ i=1 σih (xi) ] .
  Lemma 5. (Barlett & Mendelson, 2002)Let SN = {(x1, y1) , · · · , (xN , yN )} be samples chosen i.i.d. according to the distribution D. If the loss function ` is bounded by B > 0. Then for any δ ∈ (0, 1), with probability at least 1− δ, the following holds for all h ∈ H,
  R (h) ≤ RSN (h) + 2BRSN (`H) + 3B √ (log(2/δ)/(2N).
  In addition, according to the Ledoux-Talagrand inequality and assume loss function is L-lipschitz, we have
  RSN (`H) ≤ LRSN (H) .
  So the population risk can be bounded by the empirical risk and Rademacher complexity of the function class H. Because we can’t minimize the population risk directly, we can minimize it indirectly by minimizing the empirical risk and Rademacher complexity of the function class H. Next, we will further discuss Rademacher complexity of the function class H. We first introduce several lemmas below.
  Lemma 6. (Klebaner, 2005) For any given matrix U, the solution to the equation{ dx(t) = Ux(t)dt
  x(0) = x̂
  has the following expression x(t) = exp(Ut)x̂,
  Also, we can write the solution to the following equation{ dy(t) = Uy(t)dt+ γy(t)dB(t)
  y(0) = x̂
  as
  y(t) = exp(Ut− 1 2 γ2tI + γB(t)I)x̂.
  Obviously, we have E[y(t)] = x(t). Lemma 7. A matrix C ∈ Rd×d is circulant, if there exists real number a1, · · · , ad such that
  C =  a1 a2 . . . ad ad a1 . . . . . . . . . . . . . . . a2
  a2 . . . ad a1
   .
  For any circulant matrix, we have the following eigen-decomposition
  ΨHCΨ = diag(λ1, . . . , λd),
  where
  √ dΨ =  1 1 . . . 1 1 m1 . . . md−1 . . . . . . . . . . . .
  1 md−11 . . . m d−1 d−1  , and mis are the roots of unity and λi = a1 + a2mi + · · ·+ admd−1i .
  C.2 THE PROOF OF THEOREM 3
  Proof. By the definition of Rademacher complexity (Def. 2), we have
  RSN (F) = (1/N)Eσ [ sup f∈F N∑ i=1 σiwx p i (T ) ] = (c/N)Eσ [ sup ‖U‖2≤c ‖ N∑ i=1 σix p i (T ) ‖2 ] .
  Let ui = Ψx p i and denote the jth element of ui as ui,j . Then by lemma 7, we have
  RSN (F) = (c/N)Eσ sup ‖U‖2≤c ∑ i,j σiσj〈xpi (T ),x p j (T )〉 1/2
  = (c/N)Eσ sup |λi|≤c (‖ΨH N∑ i=1 σi ( ui,1 exp(λ1Tp), · · · , ui,d exp(λdTp))T ‖2 ) = (c/N)Eσ sup
  |λi|≤c { d∑ j=1 [ N∑ i=1 σiui,j exp (λjTp) ]2 }1/2 = (c/N) exp(cTp)Eσ{ d∑ j=1 [ N∑ i=1 σiui,j ]2 }1/2
  = (c/N) exp (cTp)Eσ‖ N∑ i=1 σiui‖2 = (c/N) exp (cTp)Eσ‖Ψ N∑ i=1 σix p i ‖2
  = (c/N) exp (cTp)Eσ‖ N∑ i=1 σix p i ‖2
  Note that E (wxpi (T )) = wE (x p i (T )) and according to Lemma 6, similar to proof for the function class F we have
  RSN (G) = (c/N)Eσ sup ‖U‖2≤c ∑ i,j σiσj〈Expi (T ),Ex p j (T )〉 1/2
  = (c/N)Eσ sup |λi|≤c (‖ΨH N∑ i=1 σi(ui,1 exp(λ1Tp− p(1− p)γ2T/2),
  · · · , ui,d exp(λdTp− p(1− p)γ2T/2))T ‖2)
  = (c/N)Eσ sup |λi|≤c { d∑ j=1 [ N∑ i=1 σiui,j exp ( λjTp− p(1− p)γ2T/2 )]2 }1/2 = (c/N) exp(cTp− p(1− p)γ2T/2)Eσ{ d∑ j=1 [ N∑ i=1 σiui,j ]2 }1/2 = (c/N) exp ( cTp− p(1− p)γ2T/2 ) Eσ‖
  N∑ i=1 σiui‖2
  = (c/N) exp ( cTp− p(1− p)γ2T/2 ) Eσ‖Ψ N∑ i=1 σix p i ‖2 = (c/N) exp ( cTp− p(1− p)γ2T/2 ) Eσ‖
  N∑ i=1 σix p i ‖2 < RSN (F)
  Therefore, we have completed the proof for the fact that the ensemble of Gaussian noise injected ResNets can reduce generalization error compared to the standard ResNet.
sections.heading: D EXPERIMENTS ON STRATEGY II
sections.text: D.1 FORWARD AND BACKWARD PROPAGATION USING (7)
  Figure 8 plots the forward and backward propagation of the ICLR logo using the SDE model (7). Again, we cannot simply use the backward Euler-Maruyama discretization to reverse the features generated by the propagating through the forward Euler-Maruyama discretization.
  D.2 EXPERIMENTS ON THE IDC DATASET
  In this subsection, we consider the performance of the second residual perturbation in protecting membership privacy while retaining the classification accuracy on the IDC dataset. We use the same ResNet models as that used for the first residual perturbation. We list the results in Fig. 9, these results confirm that the residual perturbation (8) can effectively protect data privacy and maintain or even improve the classification accuracy. In addition, we depict the ROC curve for this experiment in Figure 12 (c). We note that, as the noise coefficient increases, the gap between training and testing accuracies narrows, which is consistent with Theorem 3.
  D.2.1 RESIDUAL PERTURBATION VS. DIFFERENTIALLY PRIVATE STOCHASTIC GRADIENT DESCENT
  We have shown that the first residual perturbation (6) outperforms the DPSGD in protecting membership privacy and improving classification accuracy. In this part, we further show that the second residual perturbation (8) also outperforms the benchmark DPSGD with the above settings. Table 2 lists the AUC of the attack model and training & test accuracy of the target model; we see that the second residual perturbation can also improve the classification accuracy and protecting better membership privacy.
  D.3 EXPERIMENTS ON THE CIFAR10/CIFAR100 DATASETS
  In this subsection, we will test the second residual perturbation (8) on the CIFAR10/CIFAR100 datasets with the same model using the same settings as before. Figure 10 plots the performance of En5ResNet8 on the CIFAR10 dataset. These results show that the ensemble of ResNets with residual
  perturbation (8) is significantly more robust to the membership inference attack. For instance, the AUC of the attack model for ResNet8 and En5ResNet8 (γ = 2.0) is 0.757 and 0.526, respectively. Also, the classification accuracy of En5ResNet8 (γ = 2.0) is higher than that of ResNet8, and their accuracy is 71.2% and 66.9% for CIFAR10 classification. Figure 11 shows the results of En5ResNet8 for CIFAR100 classification. These results confirm that residual perturbation (8) can protect membership privacy and improve classification accuracy once more.
  D.4 ROC CURVES FOR THE EXPERIMENTS ON DIFFERENT DATASETS
  Figure 12 plots the ROC curves for the experiments on the different datasets with different models using the second residual perturbation strategy. These ROC curves again show that if γ is sufficiently large, the attack model’s prediction will be nearly a random guess.
sections.heading: E MORE EXPERIMENTAL DETAILS
sections.text: We give the detailed construction of the velocity field F (x(t),W(t)) in (3) and (7) that used to generate Figs. 1 and 8 in Algorithm 1.
  Algorithm 1 The expression of F (x(t),W(t)) Input: image=x(t); rows, cols, channels = image.shape. Output: F (x(t),W(t))=dirtyimage. for k in range(channels) do
  for i in range(rows): do for j in range(cols): do
  offset(j) =int([j + 50.0cos(2πi/180)]%cols); offset(i) =int([i+ 50.0sin(2πi/180)]%row); dirtyimage[i, j, k] = image[(i+ offset(j))%rows, (j + offset(i))%cols, k];
  return dirtyimage
sections.heading: F ARCHITECTURES OF THE USED DNS
sections.text: Figure 13 shows the architectures of ResNets used in this paper, and we plot basic blocks in Fig. 14.