summaries: In this paper, the authors propose a certain type of trainable quantum procedures used as machine learning models for the classification of images. They explicitly show how to implement the circuits for loading compressed images into quantum states that are fed to a trainable unitary that writes the predicted label into a readout register. They conclude on numerical simulation performed on the MNIST dataset.

summaries: The paper study the problem of quantum neural networks for classical image classification. It leverages the FRQI framework to enable larger size of the input image instead of using very small 4x4 input image. They construct parameterized quantum circuits using XX and ZZ gates (called "CRADL" and "CRAML") to perform the transformations of quantum state and perform classification.

summaries: The paper presents an image coding approach based on 2qubit gates. Downsampled MNIST digits are classified using standard components. Additionally, the experimental section studies the use of reduced codes.

summaries: This paper studies the image classification problem on quantum computers. The prior of image classification by Farhi et al. can only work on 4-by-4 input images, while this paper conducts experiments on 16-by-16 images for the MNIST dataset.

abstractText: Recent work suggests that quantum machine learning techniques can be used for classical image classification by encoding the images in quantum states and using a quantum neural network for inference. However, such work has been restricted to very small input images, at most 4 × 4, that are unrealistic and cannot even be accurately labeled by humans. The primary difficulties in using larger input images is that hitherto-proposed encoding schemes necessitate more qubits than are physically realizable. We propose a framework to classify larger, realistic images using quantum systems. Our approach relies on a novel encoding mechanism that embeds images in quantum states while necessitating fewer qubits than prior work. Our framework is able to classify images that are larger than previously possible, up to 16× 16 for the MNIST dataset on a personal laptop, and obtains accuracy comparable to classical neural networks with the same number of learnable parameters. We also propose a technique for further reducing the number of qubits needed to represent images that may result in an easier physical implementation at the expense of final performance. Our work enables quantum machine learning and classification on classical datasets of dimensions that were previously intractable by physically realizable quantum computers or classical simulation.

authors:
Esma Aïmeur
Gilles Brassard
Sébastien Gambs

title: Quantum speed-up for unsupervised learning

venue: Machine Learning,

authors:
Unai Alvarez-Rodriguez
Lucas Lamata
Pablo Escandell-Montero
José D. Martín-Guerrero
Enrique Solano

title: Supervised quantum learning without measurements

venue: Scientific Reports,

authors:
Platt
Chris Quintana
Eleanor G. Rieffel
Pedram Roushan
Nicholas C. Rubin
Daniel Sank
Kevin J. Satzinger
Vadim Smelyanskiy
Kevin J. Sung
Matthew D. Trevithick
Amit Vainsencher
Benjamin Villalonga
Theodore White
Z. Jamie Yao
Ping Yeh
Adam Zalcman
Hartmut Neven
John M. Martinis

title: Quantum supremacy using a programmable superconducting processor

authors:
Adriano Barenco
Charles H. Bennett
Richard Cleve
David P. DiVincenzo
Norman Margolus
Peter Shor
Tycho Sleator
John A. Smolin
Harald Weinfurter

title: Elementary gates for quantum computation

venue: Phys. Rev. A,

authors:
Kerstin Beer
Dmytro Bondarenko
Terry Farrelly
Tobias J. Osborne
Robert Salzmann
Daniel Scheiermann
Ramona Wolf

title: Training deep quantum neural networks

venue: Nature Communications,

authors:
Ethan Bernstein
Umesh Vazirani

title: Quantum complexity theory

venue: SIAM Journal on Computing,

authors:
Sergio Boixo
Sergei V. Isakov
Vadim N. Smelyanskiy
Ryan Babbush
Nan Ding
Zhang Jiang
Michael J. Bremner
John M. Martinis
Hartmut Neven

title: Characterizing quantum supremacy in near-term devices

venue: Nature Physics,

authors:
Matthias C. Caro
Ishaun Datta

title: Pseudo-dimension of quantum circuits

venue: Quantum Machine Intelligence,

authors:
Deutsch David
Jozsa Richard

title: Rapid solution of problems by quantum computation

venue: Proc. R. Soc. Lond. A,

authors:
Yuxuan Du
Min-Hsiu Hsieh
Tongliang Liu
Dacheng Tao

title: Expressive power of parametrized quantum circuits

venue: Phys. Rev. Research,

authors:
Vedran Dunjko
Jacob M. Taylor
Hans J. Briegel

title: Quantum-enhanced machine learning

venue: Phys. Rev. Lett.,

title: Classification with Quantum Neural Networks on Near Term Processors

venue: Arxiv Preprint,

title: Book review: Deep learning

venue: Healthc Inform Res,

title: A fast quantum mechanical algorithm for database search

venue: In Proceedings of the Twenty-Eighth Annual ACM Symposium on Theory of Computing,

authors:
Aram W. Harrow
John C. Napp

title: Low-depth gradient measurements can improve convergence in variational hybrid quantum-classical algorithms

venue: Physical Review Letters,

authors:
Maxwell Henderson
Samriddhi Shakya
Shashindra Pradhan
Tristan Cook

title: Quanvolutional neural networks: powering image recognition with quantum circuits

authors:
Ashish Kapoor
Nathan Wiebe
Krysta Svore

title: Quantum perceptron models

authors:
Iordanis Kerenidis
Jonas Landman
Anupam Prakash

title: Quantum algorithms for deep convolutional neural networks, 2019

authors: Corinna Cortes. MNIST handwritten digit database.

title: URL http://yann

authors:
YaoChong Li
Ri-Gui Zhou
RuQing Xu
Jia Luo
WenWen Hu

title: A quantum deep convolutional neural network for image recognition

venue: Quantum Science and Technology,

authors:
K. Mitarai
M. Negoro
M. Kitagawa
K. Fujii

title: Quantum circuit learning

authors:
Alex Monràs
Gael Sentís
Peter Wittek

title: Inductive supervised quantum learning

authors:
Michael A. Nielsen
Isaac L. Chuang

title: Quantum Computation and Quantum Information: 10th Anniversary Edition

authors:
Seunghyeok Oh
Jaeho Choi
Jong-Kook Kim
Joongheon Kim

title: Quantum convolutional neural network for resource-efficient image classification: A quantum random access memory (qram) approach

venue: International Conference on Information Networking (ICOIN),

authors:
Giuseppe Davide Paparo
Vedran Dunjko
Adi Makmal
Miguel Angel Martin-Delgado
Hans J. Briegel

title: Quantum speedup for active learning agents

venue: Phys. Rev. X,

authors:
F. Dong P.Q. Le
K. Hirota

title: A flexible representation of quantum images for polynomial preparation, image compression, and processing operations

venue: Quantum Inf Process,

title: Quantum computing and the entanglement frontier

authors:
S.E. Rasmussen
K. Groenland
R. Gerritsma
K. Schoutens
N.T. Zinner

title: Single-step implementation of high-fidelity n-bit toffoli gates

venue: Physical Review A,

authors:
Masahide Sasaki
Alberto Carlini

title: Quantum learning and universal quantum matching machine

authors:
Maria Schuld
Ilya Sinayskiy
Francesco Petruccione

title: The quest for a quantum neural network

venue: Quantum Information Processing,

authors:
Maria Schuld
Ville Bergholm
Christian Gogolin
Josh Izaac
Nathan Killoran

title: Evaluating analytic gradients on quantum hardware

authors:
Maria Schuld
Alex Bocharov
Krysta M. Svore
Nathan Wiebe

title: Circuit-centric quantum classifiers

authors:
G. Sentís
J. Calsamiglia
R. Muñoz-Tapia
E. Bagan

title: Quantum learning without quantum memory

authors:
Gael Sentís
Alex Monràs
Ramon Muñoz Tapia
John Calsamiglia
Emilio Bagan

title: Unsupervised classification of quantum data

title: Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer

title: On the power of quantum computation

authors:
Samuel L. Smith
Benoit Dherin
David G.T. Barrett
Soham De

title: On the origin of implicit regularization in stochastic gradient descent

venue: In International Conference on Learning Representations,

authors:
Kwok Ho Wan
Oscar Dahlsten
Hlér Kristjánsson
Robert Gardner
M.S. Kim

title: Quantum generalisation of feedforward neural networks. npj

venue: Quantum Information,

heading: 1 INTRODUCTION

text:
In the past decade, deep learning has been remarkably successful on a wide variety of classical learning tasks (Gi (2016); LeCun et al. (2015)). In parallel, quantum computing (QC) has long promised dramatic increases in computational power over classical computers, culminating in a recent demonstration of quantum supremacy in a machine with 53 programmable qubits in Arute et al. (2019). However, even these quantum systems are already approaching the limits of classical simulability by the world’s largest traditional supercomputers (Boixo et al. (2018); Preskill (2012)). The power of quantum computation suggests that quantum analogues of deep learning models like feedforward neural network may outperform their classical counterparts, especially when the data is inherently quantum (Wan et al. (2017); Beer et al. (2020); E. Farhi (2018)).
In this paper, we use a quantum neural network (QNN) to classify the MNIST dataset of handwritten digits (LeCun & Cortes (2010)). Prior work has been restricted to only highly-compressed, rather unrealistic input images due to their inefficient encoding schemes that are injective maps from classical images to their corresponding pure quantum states. These frameworks have used input images of a maximum resolution of 4× 4, which is too coarse even for humans to provide accurate labels (see Figure 1). For larger images, injectivity would necessitate the same number of qubits as bits that are present in the original image. However, recent advances in quantum machine learning (QML) on classical data, such as the Flexible Representation of Quantum Images (FRQI) in P.Q. Le & Hirota (2011), have demonstrated that quantum wavefunctions can utilize quantum entanglement to encode classical data using exponentially less qubits than their corresponding classical representation in bits. In this paper, we use the FRQI method to embed the input images in fewer-qubit systems. This approach necessitates a novel QNN architecture for classification, which we describe in Section 5. The input to our model are images of resolution up to 16 × 16 whose quantum encoding only requires 8 qubits (6 for pixel locations, 1 for color, and 1 for readout). To the best of our knowledge, our work is the first to propose a data encoding scheme and QNN that can be used to classify realistic images.
Our main contributions are as follows:
1. We provide a novel construction to compress images and encode them in their FRQI states. Our construction uses only 2-qubit gates, which permits its use in common quantum machine learning packages such as Cirq and Tensorflow Quantum (TFQ) and may be of independent interest (Cirq (2021); GoogleAI (2020)).
2. We propose a new QNN layers, CRADL and CRAML, which we use in a model trained with the images’ FRQI states as input.
3. We show that our trained QNN achieves accuracy comparable to classical models with the same number of parameters.
4. We propose a novel technique to further compress black and white images, and study the scaling behavior of our model with the extent of image compression.
Organization: In Section 2, we provide a brief review of the formalism of quantum computation. In Section 3, we provide an overview of related work and motivate our study. In Section 4, we describe our dataset and how each image is encoded in a quantum state. In Section 5, we describe the prescription for using a quantum neural network to obtain a classification prediction for a given image and describe the model we use for classification. In Section 6, we present our results. We conclude with Section 7 and a description of future work in Appendix 8.

heading: 2 PRELIMINARIES

heading: 2.1 QUANTUM COMPUTING

text:
Here we provide a brief review of quantum computation. For a more detailed reference and an interactive coding tutorial, we refer the reader to Nielsen & Chuang (2011) and Qiskit (2017) respectively.
In quantum computation, the basic unit of information is a two-state quantum mechanical (QM) system called a qubit; the two states are traditionally written |0〉 and |1〉. A qubit can be in either of these two states, as well as a quantum superposition of these states, formally written as a wavefunction |ψ〉 = a0 |0〉 + a1 |1〉 = ∑ i∈{0,1} ai |i〉, where each ai ∈ C. When a qubit is measured, the wavefunction collapses and the result of the measurement is state |0〉 with probability |a0|2 and state |1〉 with probability |a1|2 1. The space of all possible states of the qubit is called the Hilbert space H1; the states |0〉 and |1〉 provide a basis for this Hilbert space. 2
Multi-qubit systems are represented mathematically by the tensor product of multiple single-qubit systems. Notationally, we write
|ψN 〉 = ∑
{i1,i2,...iN}∈{0,1}N ai1,i2,...iN |i1, i2, . . . iN 〉
where the states |i1, i2, . . . iN 〉 provide a basis for the multi-qubit Hilbert space HN and |ψN 〉 is generally a superposition of these basis states. A two-qubit state |ψ〉 ∈ H2, cannot necessarily be factorized into two single-qubit states |ψ1〉 , |ψ2〉 ∈ H1:
|ψ〉 = ∑
{i1,i2}∈{0,1}2 ai1,i2 |i1, i2〉 6= |ψ1〉 ⊗ |ψ2〉 = (a0 |0〉+ a1 |1〉)⊗ (a′0 |0〉+ a′1 |1〉)
we call a state which cannot be so factored a mixed state. In particular, we notice thatH2 > H1⊗H1, and a similar result holds for multi-qubit systems with N > 2.
Under the laws of quantum mechanics, these wavefunctions – or states – evolve in time as determined by linear unitary transformations. Furthermore, any operation that is physically possible to perform
1We require |a0|2 + |a1|2 = 1 2Formally, a Hilbert space is an inner product vector space that is also a complete metric space with respect
to the distance function induced by that inner product.
on a set of qubits can be represented as a unitary operator. We refer to such unitary operators as quantum gates and note that unitary operators can be viewed as rotations in Hilbert space.
A remarkable property of quantum states is their ability to be entangled. Informally, entanglement refers to the property of quantum mechanical systems whereby the state of one qubit cannot be described independently of the other qubits’ states. For example, the state |00〉 is maximally entangled as knowledge of one qubit’s state complete specifies the other’s.
In general, quantum algorithms are procedures whereby an initial wavefunction is transformed under a sequence of unitary operations, or quantum gates, and a measurement is made of the transformed state; this measurement is often performed on a readout qubit and is the output of the algorithm. Many quantum computation algorithms are designed to exploit properties of entangled systems (Bernstein & Vazirani (1997); David & Richard (1992); Simon (1997); Shor (1997); Grover (1996)).

heading: 2.2 COMMON QUANTUM GATES

text:
Several common quantum gates are defined below. These gates are defined by their action on the basis states of the Hilbert space, since they extend linearly to superpositions of the basis states. The definitions for the single-qubit Hadamard gate H and the Pauli-X gate X are:
H |0〉 = |+〉 := 1√ 2 (|0〉+ |1〉) H |1〉 = |−〉 := 1√ 2 (|0〉 − |1〉) (1)
X |0〉 = |1〉 X |1〉 = |0〉 (2)
A common 2-qubit gate is CNOT , which flips the second qubit if the first qubit is in state |1〉:
CNOT |00〉 = |00〉 CNOT |01〉 = |01〉 (3) CNOT |10〉 = |11〉 CNOT |11〉 = |10〉

heading: 3 RELATED WORK

text:
Many studies use QNNs to model either inherently quantum or quantum-encoded classical data but are generally restricted to very small images (Li et al. (2020); Henderson et al. (2020); Oh et al. (2021)). One line of work encodes classical data in quantum systems and focuses on learning the classifier’s circuit architecture. These approaches require an injective map from the input image to a corresponding pure quantum state, which forgoes the exponential compression advantages afforded by methods such as FRQI3 (E. Farhi (2018); Aïmeur et al. (2013); Paparo et al. (2014); Schuld et al. (2014); Kapoor et al. (2016)). Amongst this line of work, E. Farhi (2018) propose the general setup that we follow in this paper. In contrast with their work, however, we use the FRQI technique to exploit the dimensionality of the multi-qubit Hilbert space and need much fewer qubits.
Other studies take the quantum wavefunction as given, either by assuming the classical data is already provided in its quantum-encoded form Schuld et al. (2020) or because they use inherently quantum data (Sasaki & Carlini (2002); Gambs (2008); Sentís et al. (2012); Dunjko et al. (2016); Monràs et al. (2017); Alvarez-Rodriguez et al. (2017); Du et al. (2020); Sentís et al. (2019); Beer et al. (2020); Caro & Datta (2020)). Amongst these papers, Schuld et al. (2020) is perhaps closest to this work. The authors, however, take the mixed-state encodings of images as given for input to a QNN and do not describe how to construct the quantum states. Other work, such as Beer et al. (2020), assumes the wavefunctions as given proposes a generalization of the perceptron to the quantum setting, which provides a more generalized framework than in E. Farhi (2018) These authors use inherently quantum data, in contrast with our work. We use classical data and explicitly construct quantum circuits to encode classical images into their wavefunctions. Our approach lends itself to direct experimentation and is usable with modern quantum machine learning packages.
Finally, a third line of work uses quantum convolutional neural networks via semi-classical simulations meant to model the noise introduced by quantum effects, as in Kerenidis et al. (2019). These approaches do not provide a fully quantum simulation to evolve the quantum states, which would require construction of the actual data’s wavefunctions as in our work.
3For a demo of a standard approach following E. Farhi (2018) using TFQ, which could also serve as a preliminary for this work, see https://www.tensorflow.org/quantum/tutorials/mnist
Throughout prior work, encoding classical data in quantum states efficiently appears to be a common open problem.

heading: 4 FORMAL SETTING

heading: 4.1 PROBLEM STATEMENT

text:
In classical image classification, the input to our model is an n× n-pixel image ∈ {0, 1}2n and our goal is to learn a classification function with binary outputs fclassical parametrized by weights w:
fclassical(w) : {0, 1}2n → {0, 1} (4)
In the quantum setting, the input to our classification function is still an n × n-pixel image but must be encoded in a dlog2ne+ 1 dimensional Hilbert spaceH by an encoding function F , where the +1 is for the readout qubit. The quantum neural network is a sequence of unitary operations U = U1◦U2◦. . . UN parametrized by angles θ = θ1, θ2, . . . , θN . To obtain a classification prediction, a measurement is performed on the readout qubit:
fquantum(θ) : {0, 1}2n → {0, 1} (5)
: {0, 1}2n F−−→ H U(θ)−−−−→ H measure−−−−→ {0, 1}
That is fquantum(θ) = measure ◦ U(θ) ◦ F . In Sections 5 and 5 we propose an implementation of the FRQI algorithm P.Q. Le & Hirota (2011) to construct F , propose a construction of U(θ), describe how to learn the parameters θ via standard backpropagation, and describe the final measurement step.

heading: 4.2 DATASET AND QUANTUM ENCODING

text:
Crucial to our approach is the encoding of a classical datapoint (e.g. an image) in a quantum state. In our experiments, we use the MNIST dataset of handwritten digits LeCun & Cortes (2010). Following E. Farhi (2018), we restrict our dataset to those of only two ground truth labels: 3 and 6. We downsample image resolutions to either 8× 8 or 16× 16 using bilinear interpolation. The remaining dataset is approximately 12, 000 training images and 1, 100 validation images for each resolution. Finally, we transform the images to black and white by thresholding the pixel color.
In Figure 1, we present an MNIST image downsampled to different resolutions. Prior work uses resolutions of only 4× 4, but loses many important features of the original data E. Farhi (2018). With the FRQI encoding and the further compression we are able to encode higher-resolution images on current quantum hardware; this insight motivates our study of different downsampled resolutions.
After preprocessing, each image is a black and white 2n × 2n = 22n dimensional binary vector. Our objective is to encode the image as a wavefunction |ψdata〉:
|ψdata〉 = ∑
q:={q0,q1,...,q2n−1}∈{0,1}2n |q0, q1, . . . , q2n−1〉 ⊗ (cos θq |0〉+ sin θq |1〉) (6)
In Equation 6, each basis state |q0, q1, . . . , q2n−1〉 of the "pixel qubits" represents a possible bitstring of length 22n with the strength of the superposition component and color determined by θq in the "color qubit". In our dataset, each θq is either 0 or π2 .
In some experiments in Section 5, we also consider allocating more qubits to encode the color angle θq instead of the pixel locations:
|ψdata〉 = ∑
q:={q0,q1,...,q2n−1}∈{0,1}2n |q0, q1, . . . , q2n−3〉 ⊗
( cos θ̃q |0〉+ sin θ̃q |1〉 )
To do this, we exploit the observation that the color qubit is always either |0〉 or |1〉 and map into it the last two pixel qubits according to the transformation:
|q2n−2, q2n−1〉 ⊗ |qc〉 → |q̃c〉 = cos θ̃q |0〉+ sin θ̃q |1〉 (7)
where each qi ∈ {0, 1} and
θ̃q = π
2
( qc +
q2n−2 2 + q2n−1 4
) = θq + π
4
( q2n−2 +
q2n−1 2
) (8)

heading: 5 METHODS

heading: 5.1 ENCODING THE IMAGES IN WAVEFUNCTIONS

text:
In our approach, we must first encode the image in a quantum wavefunction. We pass an initial state of |0 . . . 0〉 through a quantum circuit with a given structure, demonstrated by Figure 2 for a 4-qubit state. Initially, a Hadamard operationH⊗2n is performed on the 2n pixel qubits. This is followed by a series of n-controlled X-gates (also known as generalized TOFFOLI gates Toffoli (1980); Rasmussen et al. (2020); Shende & Markov (2008)) with alternating X gates that determine the color qubits which will be transformed. The n-qubit circuit is constructed recursively from smaller-qubit circuits by observing the symmetries in the construction.
To construct this circuit we need to define the generalized TOFFOLI gate for n qubits from basic two-qubit gates; this also enables our implementation in standard packages such as Cirq (Cirq (2021)) that only support backpropagation through two-qubit gates. We use the following lemma, first shown in Barenco et al. (1995), to recursively decompose the n-qubit generalized TOFFOLI gate as a sequence of (n− 1)-qubit generalized TOFFOLI gates and CNOT gates:
Lemma 1. (Barenco et al. (1995), Lemma 7.5): For a rotation matrix R(t), an n-controlled rotation gate can be decomposed into a circuit of the form shown in Figure 3.
From the recursive properties of the diagram above we see that an n-qubit controlled rotation decomposes into (2 · 3n − 1) one-qubit controlled rotations. Since we flip at most 2n pixels for a given image, we see that for each image
# one-qubit controlled gates ≤ 2n (2 · 3n − 1)
in addition to the standalone X−gates to encode the 2n × 2n image in the amplitudes of the input wavefunction. Though this bound is exponential in n, we find this acceptable as it is still classically simulable for larger images that are primarily constrained by the size of their qubit representations.

heading: 5.2 READOUT QUBIT AND PREDICTED LABELS

text:
Our wavefunction must also contain a readout qubit on which we perform measurements that will be the model’s predicted labels. As such, we prepare the wavefunction |ψin〉 = |ψdata〉 ⊗ |readout〉. We choose the Z-gate for measurement and thus initialize the readout qubit in the |+〉 state, which is common practice to produce an initially unbiased output:
|ψin〉 = |ψdata〉 ⊗H |0〉 = |ψdata〉 ⊗ |+〉 (10)
where |ψdata〉 is prepared as in Section 5.1. The model which is used to transform |ψin〉 is a QNN with L layers. Following E. Farhi (2018), each layer is represented by a parametrized unitary matrix. The model’s output state is:
|ψout(θ)〉 = U(θL) . . . U(θ1) (|ψdata〉 ⊗ |+〉) (11)
where θ := (θ1, . . . , θL). The final measurement is performed with Z−gate on the readout qubit; the predicted label is 〈ψout(θ)| I2n+1 ⊗ Z |ψout(θ)〉. We train the model’s parameters, θ1 . . . θL, via stochastic gradient descent (SGD) using these predictions and the hinge loss:
loss(i)(θ) = 1− y(i) 〈ψ(i)out(θ)| I2n+1 ⊗ Z |ψ (i) out(θ)〉 (12)
where the superscript (i) is used to refer to the ith training example.

heading: 5.3 IMPLEMENTATION DETAILS

text:
We use Cirq (Cirq (2021)) to encode the images into their respective wavefunctions and TensorFlow Quantum (TFQ) (GoogleAI (2020)) to train the model via the paradigm described in Section 5.2. TFQ permits the use of Parametrized Quantum Circuits (PQCs), which describe the unitary operations of the QNN, as a single Keras layer Chollet et al. (2015) within the standard TensorFlow framework.
Backpropagation through quantum layers is nontrivial. We recall that, for any layer, the 2n × 2n unitary operator can be expressed in terms of the exponential of a 2n × 2n Hermitian operator H called the Hamiltonian, which in turn can be decomposed into its Pauli decomposition (a tensor product of n Pauli matrices, which form an orthonormal basis over the Hilbert space of Hermitian matrices over R):
U(θ) = exp i ∑ σi∈{I,σ1,σ2,σ3}n θ(σi) ⊗ i σi  (13) When the layer is restricted to unitary operations whose Hamiltonian has a single term in the Pauli decomposition, gradients with respect to the layer’s parameters can be computed analytically using the parameter shift techniques first introduced in Mitarai et al. (2018) and Schuld et al. (2019) (Harrow & Napp (2021)). These techniques provide a way to calculate partial derivatives of parameterized quantum circuits in terms of other functions that use the same circuit architecture with shifted parameters. For this reason, we restrict the gates in our quantum layers to be multi-qubit exponential Pauli gates. For these gates, analytic gradients can also be computed because they are rotations of operations whose Hamiltonians contain a single type of term. For example, the XX-gate can be written:
(X ⊗X)θ = exp { θ ( −iπ2 (X − I)⊗−i π 2 (X − I) )} = e−i π 2 θ(X−I) ⊗ e−i π 2 θ(X−I)

heading: 5.4 NETWORK ARCHITECTURE

text:
A general 2n × 2n learnable unitary operation would consist of 22n trainable real parameters and the standard representation of this parameters follows equation equation 13. Note that, as described in Section 5.3, this does not necessarily permit analytic gradient computation in the backpropagation step.
To permit analytical gradient computations, we construct layers having a specific structure; an example such layer is demonstrated in Figure 4. Each layer consists of either XX or ZZ operators applied in succession to each pixel qubit and the readout qubit, followed by the same operation on the pixel qubit and the color qubit. Empirically, the Color-Readout-Alternating-Double-Layer architecture (CRADL) presented in Figure 4 resulted in the best performance.
Each consecutive pixel-readout and pixel-color pair of gates share the same learning parameter, a rotation angle. One double layer of the type shown in Figure 4 will have 2n learnable parameters, where n is the number of pixel qubits. A circuit with L layers will therefore have nL parameters. In the experiments in Section 6, we choose the number of layers such that the number of parameters nL is comparable to those of the classical benchmark, given a fixed number of qubits.
We note that there are equivalent network architectures that lead to comparable results, such as the Color-Readout-Alternating-Mixed-Layer architecture shown in Figure 5.

heading: 6 RESULTS

text:
We benchmark our quantum learning framework against a classical neural network with two hidden layers with ReLU activations and a single-neuron output layer; in this setting, the quantum and classical models have a comparable number of parameters. We trained the quantum neural network for 10 epochs, which is the same number of epochs after which the classical neural network began to overfit (as determined by cross-validation). All experiments were conducted on a personal laptop with no GPU (Macbook Pro, 2.4 GHz 8-Core Intel Core i9 CPU, 64 GB 2667 MHz DDR4 RAM).
Table 1 demonstrates our final results. On 8 × 8 images, the QNN without the extra two-qubit compression achieves performance comparable to the classical network, whereas the network with the extra two-qubit compression (denoted−2Q) performs worse. For 16×16 images, we add more layers to the QNN so that the the number of parameters remains comparable to the classical dense network and must use the extra two-qubit compression due to the computational cost of the experiment. We
observe that the QNN is unable to achieve the same performance as the classical network, likely due to the extensive compression of the images in the quantum states.
Figure 6 shows the validation accuracy of our quantum and baseline classical models versus training epoch. We observe that the classical neural network and both quantum neural networks demonstrate similar validation performance curves.

heading: 7 DISCUSSION AND CONCLUSION

text:
We note that the method we describe in Equation 8 to reduce the number of required qubits results in worse performance. This degradation in test accuracy may be palatable in applications that attempt to minimize their qubit usage. When we attempted to lower the number of necessary qubits further, we observed unstable learning behavior. In such settings with reduced feature dimensionality, it may be necessary to redesign the network architecture.
In this paper, we developed a proof of concept for recently proposed QNN models. In the process, we proposed a methodology to map classical images to quantum states that may be of independent interest to the community. We also propose a new form of quantum neural network layers motivated by the highly entangled input states, the CRADL and CRAML layers in Section 5.3, and demonstrate that a model consisting of these layers achieves performance competitive with classical neural networks with a comparable number of parameters. Furthermore, our work is evidence that quantum machine learning algorithms can scale to data of dimensions larger than those previously tractable by classical simulation or available quantum hardware and classify MNIST images of size 16× 16 on a personal laptop.

heading: 8 FUTURE WORK

text:
We did not do a comprehensive survey of the space of all possibly unitary operations that could be used for each hidden layer, but we could imagine that invoking 3 or more qubit gates to the layer circuits would improve the learning outcomes, as that would give more direct access to the correlation structure. The circuit above Figure 4 composed entirely of two-qubit gates is trying to work around this limitation.
We would like in the future to conduct a more systematic survey of network architectures to enhance the learning outcomes with possibly more involved quantum gates, either within TensorFlow Quantum or implemented separately, and study the cost benefit of forgoing analytic gradients for some of these gates for more flexibility in construction. Another lower hanging fruit is optimizing the encoded input circuits in terms of memory. As long as we write and simulate quantum algorithms on digital computers, representing the wavefunction in this manner seems inevitable and we should find better optimizations to describe the encoding procedure through circuits with much less gates than the nice recursive algorithms discussed in this paper.
We observed in Section 6 that compressing the images into larger quantum systems resulted in better performance at the expensive of greater complexity in physical realization. We leave a more thorough analysis of this tradeoff, including the understanding of the interplay between the data qubits and the color qubit, to future work.
We also note that we may view the encoding with limited circuit gates or on limited qubits as a form of implicit regularization, which has been observed to improve generalization performance as in Smith et al. (2016). We leave investigation of these connections to future work.
ETHICS STATEMENT
We do not foresee any ethical concerns with our work.
REPRODUCIBILITY STATEMENT
The authors have submitted all the code necessary to reproduce their results in the Supplementary Materials as a .zip. The code contains a README.md file with instructions on how to reproduce the experimental results. The only dataset required is the MNIST dataset, which is easily obtained from http://yann.lecun.com/exdb/mnist/. The preprocessing applied to the MNIST dataset is described in detail in Section 5.


summaries: This work proposes a novel framework to train a face recognition network for the federated learning setting. The proposed method leverages differentially private local clustering mechanism to allows to securelly share the information of class centers of a local client to other clients.   In addition, the consensus-aware recognition loss encourages the global consensuses for the learned face embeddings among clients and significantly improves the performance as compared with the plain federated learning baselines.

summaries: The aim of this paper is to tackle the privacy leakage issue when using federated learning technique to train deep face recognition models. The naïve practice, broadcasting the last fully connected layer, results in the leakage of ID features. But if we cut off the broadcast of $W^c$, it will cause the issue of overlapping IDs among the clients, which harms the training of face recognition model. Therefore, the authors propose their method, DPLC, to tackle this problem. The idea is to conduct online clustering, gaussian perturbation, combined supervision of cluster and local data.

summaries: The main contribution of the paper is a slight modification of the standard federated learning (FL) framework for the purpose of improving face recognition performance. Since face recognition involves learning a deep neural network for feature embedding and the weight vectors for mapping the feature embedding to an identity, the standard FL framework (e.g. FedAvg) that updates only the feature embedding network does not suffice. Hence, the paper proposes to cluster the weight vectors of each participant and transmit the cluster centers (albeit with Gaussian noise to ensure differential privacy) to the other participants. Once the cluster centers of other participants are known, the local weight vectors can be learned to avoid overlap with those received clusters.

summaries: This paper proposes a FL strategy for face recognition. Despite the wide investigation of FL, very littler research discusses the use of FL for face recognition.  The proposed of PrivacyFace can: (1) distill sanitized clusters from local class centers using Differentially Private Local Clustering (DPLC) . (2) use a consensus-aware recognition loss to  encourage global consensuses among clients, leading to a more discriminative feature learning.

abstractText: The growing public concerns on data privacy in face recognition can be greatly addressed by the federated learning (FL) paradigm. However, conventional FL methods perform poorly due to the uniqueness of the task: broadcasting class centers among clients is crucial for recognition performances but leads to privacy leakage. To resolve the privacy-utility paradox, this work proposes PrivacyFace, a framework largely improves the federated learning face recognition via communicating auxiliary and privacy-agnostic information among clients. PrivacyFace mainly consists of two components: First, a practical Differentially Private Local Clustering (DPLC) mechanism is proposed to distill sanitized clusters from local class centers. Second, a consensus-aware recognition loss subsequently encourages global consensuses among clients, which ergo results in more discriminative features. The proposed framework is mathematically proved to be differentially private, introducing a lightweight overhead as well as yielding prominent performance boosts (e.g., +9.63% and +10.26% for TAR@FAR=1e-4 on IJB-B and IJB-C respectively). Extensive experiments and ablation studies on a large-scale dataset have demonstrated the efficacy and practicability of our method.

authors:
Divyansh Aggarwal
Jiayu Zhou
Anil K Jain

title: FedFace: Collaborative learning of face recognition model

venue: arXiv preprint arXiv:2104.03008,

authors:
Fan Bai
Jiaxiang Wu
Pengcheng Shen
Shaoxin Li
Shuigeng Zhou

title: Federated face recognition

venue: arXiv preprint arXiv:2105.02501,

title: Secure face matching using fully homomorphic encryption

venue: In IEEE Conference on Computer Vision and Pattern Recognition,

authors:
George EP Box
William H Hunter
Stuart Hunter

title: Statistics for experimenters, volume 664

venue: John Wiley and sons New York,

authors:
Sumit Chopra
Raia Hadsell
Yann LeCun

title: Learning a similarity metric discriminatively, with application to face verification

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: ImageNet: A large-scale hierarchical image database

authors:
Jiankang Deng
Jia Guo
Niannan Xue
Stefanos Zafeiriou

title: ArcFace: Additive angular margin loss for deep face recognition

authors:
Mihai Dusmanu
Johannes L Schonberger
Sudipta N Sinha
Marc Pollefeys

title: Privacy-preserving image features via adversarial affine subspace embeddings

authors:
Cynthia Dwork
Jing Lei

title: Differential privacy and robust statistics

venue: In Proceedings of the forty-first annual ACM symposium on Theory of computing,

authors:
Cynthia Dwork
Aarons Roth

title: The algorithmic foundations of differential privacy

venue: Foundations and Trends in Theoretical Computer Science,

authors:
Cynthia Dwork
Frank McSherry
Kobbi Nissim
Adam Smith

title: Calibrating noise to sensitivity in private data analysis

venue: In Theory of cryptography conference,

authors:
Dan Feldman
Chongyuan Xiang
Ruihao Zhu
Daniela Rus

title: Coresets for differentially private k-means clustering and applications to privacy in mobile sensor networks

venue: In ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN),

authors:
Robin C Geyer
Tassilo Klein
Moin Nabi

title: Differentially private federated learning: A client level perspective

venue: arXiv preprint arXiv:1712.07557,

authors:
Yandong Guo
Lei Zhang
Yuxiao Hu
Xiaodong He
Jianfeng Gao

title: MS-Celeb-1M: A dataset and benchmark for large-scale face recognition

venue: In European Conference on Computer Vision,

authors:
Peter Kairouz
H Brendan McMahan
Brendan Avent
Aurélien Bellet
Mehdi Bennis
Arjun Nitin Bhagoji
Keith Bonawitz
Zachary Charles
Graham Cormode
Rachel Cummings

title: Advances and open problems in federated learning

venue: arXiv preprint arXiv:1912.04977,

authors:
Arun Kumar Jindal
Srinivas Chalamala
Santosh Kumar Jami

title: Face template protection using deep convolutional neural network

title: Concise formulas for the area and volume of a hyperspherical cap

venue: Asian Journal of Mathematics and Statistics,

authors:
Chih-Ting Liu
Chien-Yi Wang
Shao-Yi Chien
Shang-Hong Lai

title: FedFR: Joint optimization federated framework for generic and personalized face recognition

venue: arXiv preprint arXiv:2112.12496,

authors:
Weiyang Liu
Yandong Wen
Zhiding Yu
Ming Li
Bhiksha Raj
Le Song

title: Sphereface: Deep hypersphere embedding for face recognition

authors:
Guangcan Mai
Kai Cao
Xiangyuan Lan
Pong C Yuen

title: SecureFace: Face template protection

venue: IEEE Transactions on Information Forensics and Security,

authors:
Brianna Maze
Jocelyn Adams
James A Duncan
Nathan Kalka
Tim Miller
Charles Otto
Anil K Jain
W Tyler Niggel
Janet Anderson
Jordan Cheney

title: IARPA Janus benchmark-C: Face dataset and protocol

venue: In International Conference on Biometrics,

authors:
Brendan McMahan
Eider Moore
Daniel Ramage
Seth Hampson
Blaise Aguera y Arcas

title: Communication-efficient learning of deep networks from decentralized data

venue: In Artificial Intelligence and Statistics,

authors:
H Brendan McMahan
Daniel Ramage
Kunal Talwar
Li Zhang

title: Learning differentially private recurrent language models

venue: arXiv preprint arXiv:1710.06963,

authors:
Qiang Meng
Xiaqing Xu
Xiaobo Wang
Yang Qian
Yunxiao Qin
Zezheng Wang
Chenxu Zhao
Feng Zhou
Zhen Lei

title: Poseface: Pose-invariant features and pose-adaptive loss for face recognition

venue: arXiv preprint arXiv:2107.11721,

authors:
Qiang Meng
Chixiang Zhang
Xiaoqiang Xu
Feng Zhou

title: Learning compatible embeddings

venue: In International Conference on Computer Vision,

authors:
Qiang Meng
Shichao Zhao
Zhida Huang
Feng Zhou

title: MagFace: A universal representation for face recognition and quality assessment

authors:
Kobbi Nissim
Uri Stemmer
Salil Vadhan

title: Locating a small cluster privately

venue: In ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems,

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

venue: arXiv preprint arXiv:1511.06434,

authors:
Rajeev Ranjan
Carlos D Castillo
Rama Chellappa

title: L2-constrained softmax loss for discriminative face verification

venue: arXiv preprint arXiv:1703.09507,

authors:
Florian Schroff
Dmitry Kalenichenko
James Philbin

title: FaceNet: A unified embedding for face recognition and clustering

authors:
Yantao Shen
Yuanjun Xiong
Wei Xia
Stefano Soatto

title: Towards backward-compatible representation learning

authors:
Reza Shokri
Vitaly Shmatikov

title: Privacy-preserving deep learning

venue: In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security,

title: Improved deep metric learning with multi-class n-pair loss objective

venue: In Annual Conference on Neural Information Processing Systems,

authors:
Uri Stemmer
Haim Kaplan

title: Differentially private k-means with constant multiplicative error

authors:
Stacey Truex
Nathalie Baracaldo
Ali Anwar
Thomas Steinke
Heiko Ludwig
Rui Zhang
Yi Zhou

title: A hybrid approach to privacy-preserving federated learning

venue: In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security,

authors:
Paul Voigt
Axel Von dem Bussche

title: The EU general data protection regulation (GDPR)

venue: A Practical Guide, 1st Ed., Cham: Springer International Publishing,

authors:
Hao Wang
Yitong Wang
Zheng Zhou
Xing Ji
Dihong Gong
Jingchao Zhou
Zhifeng Li
Wei Liu

title: CosFace: Large margin cosine loss for deep face recognition

authors:
Mei Wang
Weihong Deng

title: Mitigating bias in face recognition using skewness-aware reinforcement learning

authors:
Mei Wang
Weihong Deng
Jiani Hu
Xunqiang Tao
Yaohai Huang

title: Racial faces in the wild: Reducing racial bias by information maximization adaptation network

authors:
Cameron Whitelam
Emma Taborsky
Austin Blanton
Brianna Maze
Jocelyn Adams
Tim Miller
Nathan Kalka
Anil K Jain
James A Duncan
Kristen Allen

title: IARPA Janus benchmark-B face dataset

venue: In IEEE Conference on Computer Vision and Pattern Recognition Workshop,

authors:
Bin Xiao
Jiannong Cao
Qingfeng Zhuge
Yi He
Edwin H.-M Sha

title: Approximation algorithms design for disk partial covering problem

venue: In 7th International Symposium on Parallel Architectures, Algorithms and Networks,

authors:
Xiaqing Xu
Qiang Meng
Yunxiao Qin
Jianzhu Guo
Chenxu Zhao
Feng Zhou
Zhen Lei

title: Searching for alignment in face recognition

venue: In AAAI Conference on Artificial Intelligence,

authors:
Kaiyu Yang
Jacqueline Yau
Li Fei-Fei
Jia Deng
Olga Russakovsky

title: A study of face obfuscation in ImageNet

venue: arXiv preprint arXiv:2103.06191,

authors:
Dong Yi
Zhen Lei
Shengcai Liao
Stan Z Li

title: Learning face representation from scratch

venue: arXiv preprint arXiv:1411.7923,

heading: 1 INTRODUCTION

text:
Face recognition technique offers great benefits when used in right context, such as public safety, personal security and convenience. However, misuse of this technique is a concern as it involves unique and irrevocable biometric data. The rapid commercial applications based on face recognition and facial analysis techniques have stimulated a global conversation on AI ethics, and have resulted in various actors from different countries issuing governance initiatives and guidelines. EU’s General Data Protection Regulation (GDPR) (Voigt & Von dem Bussche, 2017), California Consumer Privacy Act (CCP) and Illinois Personal Information Protection Act (IPI) enforces data protection “by design and by default” in the development of any new framework. On the nationwide “315 show” of year 2021, the China Central Television (CCTV) called out several well-known brands for illegal face collection without explicit user consent. As researchers, it is also our duty to prevent the leakage of sensitive information contained in public datasets widely used by the research community. Therefore, faces in ImageNet (Deng et al., 2009) were recently all obfuscated (Yang et al., 2021) and a large face dataset called MS-Celeb-1M (Guo et al., 2016) was pulled of the Internet.
In the wake of growing social consensus on data privacy, the field of face recognition calls for a fundamental redesign about model training while preserving privacy. A potential solution is the paradigm called Federated Learning (FL) (McMahan et al., 2017a). Given C clients with local datasets {D1,D2, · · · ,DC} as shown in Fig. 1a, FL decentralizes the training process by combining local models fine-tuned on each client’s private data and thus hinders privacy breaches. Typical examples of these clients include personal devices containing photo collections of a few family members, or open-world scenarios such as tourist attractions visited by tens of thousands of people. In most circumstances, we can safely assume very few classes would co-exist in two or more clients.
Despite the numerous FL-based applications in various domains (Kairouz et al., 2019) ranging from health to NLP, there are very little progress (Aggarwal et al., 2021; Bai et al., 2021; Liu et al., 2021) in training face recognition models with FL schemes. Unlike other tasks, parameters of the last classifier for a face recognition model are crucial for recognition performance but strongly associated with privacy. These parameters can be regarded as mean embeddings of identities (Wang et al., 2018;
Meng et al., 2021b; Shen et al., 2020) (also called as class centers), from where individual privacy could be spied out as studied by plenty of works (Kumar Jindal et al., 2018; Boddeti, 2018; Mai et al., 2020; Dusmanu et al., 2021). That prevents the FL approach from broadcasting the whole model among clients and the central server, and consequently leads to conflicts in the aggregation of local updates. As depicted in the global feature distribution of Fig. 1a, both clients try to spread out their own classes in the same area (pointed by the arrow) of the normalized feature space. Thus, the training loss could oscillate to achieve consensus given the sub-optimal solutions from multiple clients. On the other hand, a large batch with sufficient negative classes is necessary to learn a discriminative embedding space for advanced face recognition algorithms. During the conventional FL updates, each class is only aware of local negative classes while those from other clients are untouchable. This further limits performances of FL approaches in face recognition.
The privacy-utility paradox motivates us to introduce PrivacyFace, a framework improves federated learning face recognition by broadcasting sanitized information of local class globally. In the framework, a novel algorithm called Differentially Private Local Clustering (DPLC) first generates privacy-agnostic clusters of class centers while any specific individual in the cluster cannot be learned, irrespective of attacker’s prior knowledge, information source and other holds. Recall that the privacy cost of a differential privacy scheme is propotional to the l2-sensitivity while inversely propotional to the query number. Our DPLC reaches a low l2-sensitivity by restricting the cluster size as well as covering sufficient class centers. In addition, the number of necessary centers to communicate in DPLC is irrelevant to the number of classes in the training data. These characteristics jointly equip DPLC with much smaller privacy cost than the naive alternative, which sanitizes each class center individually by Gaussian noise. In our experiments, DPLC’s privacy cost is only 1.7e-7 of that of the naive approach. That persuasively reveals the high security level of our approach.
The second part of PrivacyFace is the consensus-aware face recognition loss. Following principles of Federated Averaging (FedAvg) (McMahan et al., 2017a), a server iteratively gathers feature extractors and privacy-agnostic clusters from clients, averages parameters of feature extractors and distributes them to clients. Accordingly, the consesus-aware loss notifies each client not to embed samples in the inappropriate zone (differential private clusters marked by DP) of the feature space during the local optimization, as shown in Fig. 1b. This process aids each client to train more discriminative features as well as align all consensuses. Compared to the conventional approach, our PrivacyFace boosts performances by +9.63% and +10.26% for TAR@FAR=1e-4 on IJB-B and IJBC respectively with only single-digit privacy cost. Moreover, the additional computational cost as well as communication cost are negligible (e.g., the extra clusters to broadcast only occupy 16K storage while the backbone already takes 212M). In a word, PrivacyFace is an efficient algorithm which improves conventional federated learning face recognition by a large margin on performances, while requires little privacy cost as well as involves lightweight computational/communication overheads.

heading: 2 PRELIMINARIES

heading: 2.1 DEEP FACE RECOGNITION

text:
Most of early works in deep face recognition rely on metric-learning based loss, including contrastive loss (Chopra et al., 2005), triplet loss (Schroff et al., 2015) and N-pair loss (Sohn, 2016). These methods are usually inefficient in training on large-scale datasets. One possible reason is that their embedding spaces at each iteration are constructed only by a positive sample and limited negative ones. Therefore, the main body of research (Ranjan et al., 2017; Liu et al., 2017; Wang et al., 2018; Deng et al., 2019; Xu et al., 2021; Meng et al., 2021a;c) has focused on devising more effective classification-based loss and achieved leading performances on a number of benchmarks. Suppose that we are given a training dataset D with N face samples {xi, yi}Ni=1 of n identities, where each xi is a face image and yi ∈ {1, · · · , n} denotes its associated class label. Considering a feature extractor φ generating the embedding fi = φ(xi) and a classifier layer with weights W = [w1, · · · ,wn], a series of face recognition losses can be summarized as
Lcls(φ,W) = − N∑ i=1 log eu(wyi ,fi) eu(wyi ,fi) + ∑n j=1,j 6=yi e v(wj ,fi) . (1)
The specific choices of the similarity functions u(w,f), v(w,f) yield different variants, e.g.:
CosFace (Wang et al., 2018) : u(w,f) = s · (cos θ −m), v(w,f) = s · cos θ, (2) ArcFace (Deng et al., 2019) : u(w,f) = s · cos(θ −m), v(w,f) = s · cos θ, (3)
where s,m are hyper-parameters and θ = arccos(wTf/‖wTf‖) is the angle between w and f .

heading: 2.2 DIFFERENTIAL PRIVACY

text:
Differential Privacy (DP) (Dwork et al., 2006) is a well-established framework under which very little about any specific individual can be learned in the process irrespective of the attacker’s prior knowledge, information source and other holds (Feldman et al., 2017; Nissim et al., 2016; Stemmer & Kaplan, 2018). We state the relevant definitions and theories (Dwork & Roth, 2014) below. Definition 1 (Differential Privacy). A randomized algorithm M : X → Y is ( , δ)-DP if for every pair of neighboring datasets X,X ′ ∈ X (i.e., X and X ′ differ in one row), and every possible output T ∈ Y the following equality holds: Pr[M(X) ∈ T ] ≤ e Pr[M(X ′) ∈ T ] + δ.
Here , δ ≥ 0 are privacy loss parameters, which we consider a privacy guarantee meaningful if δ = o( 1n ), where n is the size of dataset.
Definition 2 (l2-sensitivity). The l2-sensitivity of a function f : X → Rd is ∆2(f) = max
X,X′∈X X,X′are neighbors
‖f(X)− f(X ′)‖2. (4)
Definition 3 (Gaussian Mechanism). Suppose a function f : X → Rd and Id is the ddimensional identity matrix. The mechanism M(X) = f(X) + N (0, σ2Id) is ( , δ)-DP if σ ≥ ∆2(f) √ 2 ln( 1.25δ ).
Definition 4 (Composition of Differentially Private Algorithms (Dwork & Roth, 2014; Dwork & Lei, 2009)). Suppose M = (M1,M2, · · · ,Mk) is a sequence of algorithms, where Mi is ( i, δi)differentially private, and the Mi’s are potentially chosen sequentially and adaptively. Then M is ( ∑k i=1 , ∑k i=1 δk)-differentially private.

heading: 3 PRIVACYFACE

text:
This section details the PrivacyFace framework as illustrated in Fig. 2. At its core is a novel clustering algorithm that extracts non-sensitive yet informative knowledge about the local class distribution (Fig. 2a). After the central server broadcasting local DP-guaranteed outputs (Fig. 2b), each client optimizes over a consensus-aware objective that takes into account both the local data and the privacy-agnostic clusters to learn a discriminative embedding space (Fig. 2c) for face recognition.
p
local class centers
p
local class centers
p
local class enters

heading: 3.1 DIFFERENTIALLY PRIVATE LOCAL CLUSTERING

text:
Our target is to improve performances of federated learning face recognition by communicating auxiliary and privacy-agnostic information among clients. To distill useful information from each client and resolve the privacy-utility paradox, we propose a specialized algorithm called Differential Private Local Clustering (DPLC) with rigorous theoretical guarantees.
Problem Design. To facilitate the discussion, let us first introduce the margin parameter ρ ∈ [0, π], which defines the boundary of a cluster centered at p by saying that any w in the cluster satisfies1 arccos(wTp) ≤ ρ. We design the local clustering problem with two principles. First, the clustering results are expected to carry information only about population rather than individual from local dataset. Supported by Theorem 2 proved later, this principle ensures PrivacyFace to gain insight about the underlying distribution, while preserving the privacy of individual record. Second, the spaces confined by the clusters should be tight because they represent the inappropriate zones to escape from in other clients’ perspectives. A cluster with large margin ρ would occupy a huge proportion of the sphere, leaving limited room for other embeddings (e.g., half of sphere would be occupied if ρ = π/2). More specifically, we quantify the occupancy ratio by the following theorem: Theorem 1. Assume there is a unit and d-dimensional sphere Sd and an embedding point p ∈ Sd. Embeddings with angles less than ρ to p (i.e., {f : arccos(fTp) ≤ ρ}) occupy 12Isin2(ρ) ( d−1 2 , 1 2
) of the surface area of Sd. Here Ix(a, b) is the regularized incomplete beta function.
Proved in Sec. A.1.3 of the appendix, Theorem 1 indicates that the occupancy ratio increases monotonically with respect to the margin ρ given a feature dimension d. Fig. 3 plots a series of functions between cluster margin ρ and occupancy ratio under different dimensions. Take the rightmost curve corresponding to a typical setting d = 512 in face recognition as an example. The occupancy ratios are 0.055, 5 · 10−5, 4 · 10−10 when ρ = 1.5, 1.4, 1.3 respectively. In another word, the whole feature space would be fully occupied if we sample 20, 000 clusters over the original face classes when ρ = 1.4. It is obvious
that the more space left for optimization, the higher likelihood to achieve better performance. Therefore, we seek for a margin-constrained clustering of local data with fixed ρ ≤ 1.4 when d = 512 for better privacy-utility trade-offs. A counter example to the latter principle is the classical k-means algorithm, which have been extended to differentially private versions in a number of works (Nissim et al., 2016; Feldman et al., 2017; Stemmer & Kaplan, 2018). However, k-means naturally has no control about the scopes of the generated clusters, not to mention the difficult hyper-parameter k to pick. That motivates us to propose a specified clustering algorithm for our task.
1We adopt cosine similarity by assuming all vectors lie on a unit sphere in face recognition.
An Approximated Solution. Given the margin ρ, the problem of finding clusters to cover all class centers is a generalization of the well-studied 2D disk partial covering problem which is unfortunately NP-complete (Xiao et al., 2004). We take a greedy approach by iteratively finding a cluster centered at p∗ with margin ρ to cover the most class centers wi and weeding out those covered before the next round. We cast the problem of identifying such a cluster at each iteration as: Definition 5 (Spherical Cap Majority Covering Problem). Denote 1(·) as an indicator function. Assuming a unit d-dimensional sphere Sd and class centers {w1,w2, · · · ,wn} ∈ Sd, the target is to find a cluster center p∗ ∈ Sd where p∗ = arg maxp∈Sd ∑n i=1 1(arccos(w T i p) ≤ ρ).
Finding the optimal p∗ is also NP-complete. To address this, we first sort out the densest area S in terms of the neighbor count for eachw. An efficient approximation of p∗ is then the average of class centers in the area, p = 1|S| ∑ i∈S wi. Lines 3-5 of Algorithm 1 summarize the detailed procedure, by which the approximate cluster centers pi can be quickly enumerated as depicted in Fig. 2a. In line 8, the class centers would be further normalized in accordance with the DP mechanism.
Algorithm 1: Differentially Private Local Clustering (DPLC) Data: Class center embeddings W = {w1,w2, · · · ,wn} encoded in the classifier layer. Parameters: Margin ρ; Minimum cluster size T ; Maximum #queries Q; Privacy budget , δ. Result: Privacy-agnostic cluster centers p̂’s. Let I = {1, 2, · · · , n} and calculate θi,j = arccos(wTi wj) for i, j ∈ {1, 2, · · · , n};1 for q = 1, 2, · · ·Q do2 For each {wi, i ∈ I}, find the indexes of its neighbors by Si = {j : θi,j ≤ ρ, j ∈ I};3 Find a set with the most elements, i.e., S = arg max |Si| ;4 p← 1|S| ∑ i∈S wi ; // approximate solution for problem 55 if |S| ≥ T then6 Sample a v from distribution N (0, σ2Id) where σ = 2|S| √ (1− cos(2ρ)) ln(1.25δ ) ;7
p̂q ← p+v‖p+v‖ , I ← I/{arccos(w T i p ‖p‖2 ) ≤ ρ, i ∈ I} ;8
else9 Break out of the loop ;10
Differential Privacy Endorsement. Although the centers p reveal the population-level property of the local training set, they are still outcomes of a deterministic algorithm, thereby vulnerable to adversary attack during the FL updating. We prove below that p can be perturbed to achieve DP: Theorem 2. Define a function p , f(S) = 1|S| ∑ i∈S wi. Then the Gaussian Mechanism p̂ ,
M(S) = f(S) +N (0, σ2Id) is ( , δ)-DP if σ ≥ 2|S|· √ (1− cos(2ρ)) ln(1.25δ ).
Proof. Let S be the set of indexes of class centers which have cosine similarities larger than cos ρ with respect to a centerwo. Assuming that S,S ′ are neighbors differed atw andw′, where vectors are normalized, i.e., ‖w‖ = ‖w′‖ = ‖wo‖ = 1, then we have wTo w ≥ cos ρ and wTo w′ ≥ cos ρ.
Lemma 2 in appendix states that w Tw′ ‖w‖2‖w′‖2 ≥ cos(arccos wTo w ‖wo‖2‖w‖2 + arccos wTo w
′
‖wo‖2‖w′‖2 ) for all {w,w′,wo}. Therefore, the lower bound of wTw′ is
wTw′ = wTw′
‖w‖2‖w′‖2 ≥ cos(arccos w
T o w
‖wo‖2‖w‖2 + arccos
wTo w ′
‖wo‖2‖w′‖2 )
≥ cos(ρ+ ρ) = cos(2ρ) Following this inequality, the l2-sensitivity of f(S) is 1|S| √
2− 2 cos(2ρ) as ‖f(S) − f(S ′)‖2 = 1 |S|‖w−w ′‖2 = 1|S| √ 2− 2wTw′ ≤ 1|S| √ 2− 2 cos(2ρ). By Definition 3, we easily conclude that
M(S) = 1|S| ∑ i∈S wi +N (0, σ2Id) is ( , δ)-DP if σ ≥ 2 |S|· √ (1− cos(2ρ)) ln(1.25δ ).
Additional proofs can be found in Sec. A.1.2 of appendix. Theorem 2 outlines the setting of noise variance σ given the privacy budget , δ and the cluster margin ρ. Basically, the lower bound of
σ is proportional to 1|S| . That implies when the size of cluster |S| is large enough, adding small noise is sufficient to achieve promising privacy level. We found setting minimum cluster size to T , min |S| = 512 yields empirically stable performance. It is worth emphasizing that the number of original classes n does not directly influence the introduced noise σ. This advantage ensures that DPLC maintains a high utility when dealing with large-scale face recognition problems.
Full algorithm. Algorithm 1 presents the full DPLC and we highlight several key properties below. Properties 1, 2 state that DPLC is efficient and differentially private in theory. Besides the efficency and privacy, another major concern of the DP algorithm is the utility. Specifically, as p̂ is a combination of the source cluster center p and Gaussian noise v, we should prevent the noise v to have overwhelming effects to the output, which is further verified by our Property 3 and Property 4. Property 1. The complexity of the DPLC algorithm is O(Q · n2).
Proof. The computational complexity of calculating θi,j is O(n2). In each loop, we also compare the value θi,j ,∀i, j ∈ S with ρ, whose worst time complexity is O(n2). Because remaining steps are of linear complexity, the total complexity is O(n2 +Q · n2) = O(Q · n2).
Property 2. DPLC is a (Q · ,Q · δ)-differentially private algorithm.
Proof. The mechanism f(S) + N (0, σ2Id) is ( , δ)-differentially private according to Lemma 2. As DPLC queries at most Q results, it’s easy to conclude that our DPLC algorithm is (Q · ,Q · δ)differentially private based on Definition 4.
Property 3. Denote Φ as the cumulative distribution function of a standard Gaussian distribution. For vector magnitudes of p,v, we have ‖p‖2 ∈ (cos ρ, 1] and P (‖v‖2 ≤ r) ' Φ( r 2
σ2 √ 2(d−1) − d−12 ).
Proof. For p, the upper bound is 1 as ‖p‖2 = ‖ 1|S| ∑ i∈S wi‖2 ≤ 1 |S| ∑ i∈S ‖wi‖2 = 1. As S denotes the set of indexes of class centers which has cosine similarities larger than cos ρ with a center wo, the lower bound of ‖p‖2 can be calculated by
‖p‖2 = ‖p‖2‖wo‖2 ≥ pTwo = ( 1 |S| ∑ i∈S wi) Two = 1 |S| ∑ i∈S wTi wo ≥ 1 |S| (1 + (|S| − 1) cos ρ) > cos ρ.
Therefore, we have ‖p‖2 ∈ (cos ρ, 1]. For v ∼ N (0, σ2Id), let v = [v1, v2, · · · , vd], we have viσ following the standard normal distribution for all i. Thus, the sum of square of viσ follows a χ
2-distribution with d− 1 degrees of freedom, i.e., 1 σ2 ‖v‖ 2 2 = ∑d i=1 ( vi σ )2 ∼ χ2d−1. The mean and variance of ‖v‖22 are E(‖v‖22) = σ2(d − 1) and V ar(‖v‖22) = 2σ4(d− 1), based on properties of the χ2-distribution. By the central limit theorem, ‖v‖22 converges toN (σ2(d− 1), 2σ4(d− 1)) when d is large enough. In practice, for d ≥ 50, ‖v‖22 is sufficiently close to N (σ2(d− 1), 2σ4(d− 1)) (Box et al., 1978). Therefore, we have
P (‖v‖2 ≤ r) = P ( ‖v‖22 − σ2(d− 1)√
2σ4(d− 1) ≤ r 2 − σ2(d− 1)√ 2σ4(d− 1) ) ' Φ( r 2 σ2 √ 2(d− 1) − √ d− 1 2 ).
Note that in our work, σ can be a small value with little privacy cost. Thus, with high probability, the norm of v is close to zero. Property 4. If ‖p‖2 ≥ ‖v‖2, the cosine similarity between p and p̂ is always greater than√ 1− ‖v‖22/|p‖22.
Proof. Denote magnitudes of p,v as lp, lv and cosine similarity between p and v as x (i.e., x = p·v ‖p‖‖v‖ ∈ [−1, 1]). Let a = lr lp < 1 and s be the cosine similarity between p + v and p, then s = (p+v) Tp
‖p+v‖2‖p‖2 = l2p+lplvx lp √ l2p+l 2 v+2lplvx = 1+ax√ 1+a2+2ax . The first order derivative of s with respect to x
is dsdx = a2(a+x)
(1+a2+2ax) 3 2
. It’s easy to conclude that s takes the smallest value when x = −a, which can be reached as a is always smaller than 1. In the end, we have s ≥ √
1− l2v/l2p.
In DPLC, we reasonably require large local clusters S at line 6 while preserving privacy in terms of , δ. That constraint ensures the synthesized noise v to have a small magnitude (property 3), which thereby results in slight offsets of the sanitized vector p̂ to the source p (property 4). With these theoretical guarantees, DPLC can obtain good end-to-end utility with low privacy cost.

heading: 3.2 OPTIMIZATION WITH GLOBAL CONSENSUS

text:
PrivacyFace employs a Federated Learning (FL) paradigm for optimization. Suppose there are C clients, each of which c = 1, · · · , C owns a training dataset Dc with Nc images from nc identities. For client c, its backbone is parameterized by φc and the last classifier layer encodes class centers in Wc = [wc1, · · · ,wcnc ]. Without loss of generality, we assume that each class center w c is normalized. Central to PrivacyFace is the DPLC algorithm, which generates for each client, Qc clusters defined by the centers Pc = {p̂c1, · · · , p̂cQc} with margin ρ. As presented in Algorithm 2, the training alternates the following updates in a sufficient number of rounds M .
Client-side. In line 3 of Algorithm 2, the server broadcasts current feature extractor φt−1 to each client. Then each client updates its local cluster centers Pc that can be safely shared to others in line 5. After receiving P = {P1, · · · ,PC}, each client executes one-round optimization over the consensus-aware face recognition loss on local dataset Dc in line 7:
Lc(φc,W c) = − Nc∑ i=1 log e u(wyc i ,fci ) e u(wyc i ,fci ) + ∑nc j=1,j 6=yci ev(wj ,f c i ) + ∑C k=1,k 6=c ∑Qk l=1 e µ(p̂k l ,fci ,ρ) , (5)
where f ci is the feature extracted by φ c on i-th instance in client c. µ(p̂,f , ρ) = s · cos(max(θ − ρ, 0)) computes the similarity between f and the cluster centered at p̂ with margin ρ. As illustrated in Fig. 2c, PrivacyFace aims to learn a globally consistent embedding that can not only classify local classes, but also achieve consensuses with other clients on the incompatible clusters p̂.
Server-side. Similar to other FL approaches, a central server orchestrates the training process and receives the contributions of all clients to the new feature extractor at line 8. The well-known FedAvg (McMahan et al., 2017a) is then utilized to compute an average of all local models. Compared to the conventional FL method as described in Fig. 1a, PrivacyFace empirically achieves better convergence thanks to the consensus-aware loss that implicitly takes the data distribution of other clients into account. Built on the client-wise DPLC algorithm, the framework is potentially compatible with other optimizer, e.g., FedSGD (Shokri & Shmatikov, 2015) as shown in Sec. A.2.1 of appendix.
Algorithm 2: The PrivacyFace Training Scheme. Data: Dcs exclusively owned by each of the C clients; A pre-trained extractor φ0. Parameters: DPLC-related ρ, T,Q; Privacy budget , δ; Maximum #communications M . Result: A general recognition model φM . for t = 1, · · · ,M do1 for each client c = 1, · · · , C do2 Synchronize the local extractor φct−1 = φt−1 as the up-to-date one in server;3 Generate privacy-agnostic Pc = {p̂c1, p̂c2, · · · , p̂cqc} via Algorithm 1 ;4
The server gathers Pc and distributes P = {P1,P2, · · · ,PC} to all clients;5 for each client c = 1, · · · , C do6 Update local model {φct−1,W ct−1} to {φct ,W ct } by optimizing the loss (Eq. 5) ;7 Communicate the feature extractor φct to the server while keepingW c t locally;8
The server updates the model by φt = 1C ∑C c=1 φ c t ;9

heading: 4 EXPERIMENTS

text:
This section together with the appendix describes extensive experiments on challenging benchmarks to illustrate the superiority of PrivacyFace in training face recognition with privacy guarantee.
Datasets. CASIA-WebFace (Yi et al., 2014) contains 0.5M images from 10K celebrities and serves as the dataset for pre-training. BUPT-Balancedface (Wang & Deng, 2020), which comprises four
sub-datasets categorized by racial labels (including African, Asian, Caucasian and Indian) and each sub-dataset contains 7K classes and 0.3M images, is used to simulate the federated setting. We adopt RFW (Wang et al., 2019), IJB-B (Whitelam et al., 2017) and IJB-C (Maze et al., 2018) for evaluation. RFW is proposed to study racial bias and shares the same racial taxonomy as BUPT-Balancedface. IJB-B and IJB-C are challenging ones, containing 1.8K and 3.5K subjects respectively from largevolume in-the-wild images/videos. All images are aligned to 112× 112 based on five landmarks. Training. We assign one client for each of the four sub-datasets and use the perfect federated setting (i.e., no client goes offline during training). For the pre-trained model φ0, we adopt an opensource one2 trained on CASIA-WebFace, which builds on ResNet18 and extracts 512-d features. We finetune φ0 by SGD for M = 10 communication rounds on BUPT-Balancedface, with learning rate 0.001, batch size 512 and weight decay 5e-4. For reproducibility and fair comparison, all models are trained on 8 1080Ti GPUs with a fixed seed. To alleviate the large domain gaps across subdatasets, we build a lightweight public dataset with the first 100 classes from CASIA-WebFace to finetune local models before gathered by the server. Unless stated otherwise, the parameters for PrivacyFace are default to T = 512, Q = 1, ρ = 1.3 and = 1. As the parameter δ is required to be O( 1|D| ) (Dwork & Roth, 2014), we set δ = 1 |D|1.1 ≈ 5e-5 in BUPT-Balancedface.
Baselines. In the absence of related methods, we compare PrivacyFace with or without Gaussian noise added, and against a conventional FL method as indicated in Fig. 1a. We denote these methods as φ+ p̂, φ+p and φ respectively, based on communicated elements among the server and clients. Besides, we implement centralized training on the global version of BUPT-Balancedface and denote the trained model as “global training”. The model is also finetuned from φ0 for 10 epochs with learning rate of 0.001 and serves as an upper bound in all experiments.
Privacy-Utility Trade-Offs of the DPLC Algorithm. The goal of DPLC is to broadcast discriminative yet privacy-preserving knowledge through a federated network among clients. This part investigates its effectiveness in terms of similarities between the noise-free center p and the perturbed one p̂ by the Gaussian Mechanism with respect to different parameters. We first derive the class centersW from the Caucasian sub-dataset of BUPT-Balancedface by φ0. TakingW as input, DPLC generates 1000 p̂′s for each p, yielding the distribution of cosine similarities shown in Fig. 4. As a common sense, two face features holding cosine similarity over 0.7 are recognized as from the same identity with a high probability. Fig. 4a reveals that ρ > 1.2 can always lead to accurate p̂ with = 1 and Q = 1. If {ρ,Q} = {1.3, 1}, we conclude that p̂,p are similar with privacy cost over 0.3, as indicated by Fig. 4b. Although Q = 1 leads to good recognition performance, Fig. 4c studies the effect when more queries are required (Q = 3) at different cost . At ρ = 1.3, we can generate three clusters with descending sizes (1773, 743, 581) w.r.t the query index. By choosing a strict privacy cost = 0.33, queries 2 and 3 are too noisy to carry useful information. Setting = 1 to a reasonable privacy level, however, all queries will convey accurate descriptions of features.
Ablation Studies on Hyper-parameters. We conduct several ablation studies on PrivacyFace and present recognition performances on IJB-C in Fig. 5. Models are trained by ArcFace with default parameters if not specifically stated. In Fig. 5a, poor recognition performance and unstable training process can be observed for conventional FL method φ mainly due to the insufficient number of local classes and inconsistent stationary points achieved by different clients. These drawbacks can be significantly relieved by PrivacyFace φ + p̂. Fig. 5b reveals that performances of PrivacyFace improves as privacy cost increases. The performance is nearly saturated when > 0.3, closing to
2https://github.com/IrvingMeng/MagFace.
the method φ + p without privacy protection. This end-to-end evidence indicates DPLC achieves high privacy-utility trade-offs as analyzed in Fig. 4b. The effect of cluster margin ρ is explored in Fig. 5c and the optimal ρ is around 1.3. A small ρ leads to over-fined clusters with inadequate class centers, adding requirements to increase noise and hurt recognition. Alternatively, the performance would drop by adopting a large ρ which generates trivial clusters with high occupancy ratio.
Performances on Benchmarks. Tab. 1 presents verification performances on various benchmarks. By finetuning on BUPT-Balancedface,φ,φ+p̂ andφ+p all achieve performance boosts. Compared to the conventional FL method (φ), performances of our PrivacyFace (φ+ p̂) are consistently higher regardless of which training loss used. Specifically, improvements on TAR@FAR=1e-4 on IJB-B and IJB-C are 3.54% and 3.56% with CosFace, and 9.63% and 10.26% with ArcFace, which are significant and demonstrate the superiority of our method. We also observe that φ + p̂ and φ + p achieve very close results in all benchmarks. That implies the robustness of the proposed method.
Additional experiments can be found in appendix. We implement FedSGD (Shokri & Shmatikov, 2015) in Sec. A.2.1 to show the scalability of the PrivacyFace. Sec. A.2.2 compares the DPLC with a naive approach while Sec. A.2.3 discusses the necessity of FL methods in federated setting. In Sec. A.2.4, we analyze experimental attacks to further verify privacy guarantees of our framework.
Cost Analysis. PrivacyFace introduces little computational cost thanks to the efficient DPLC algorithm as well as the consensus-aware loss. Apart from the backbone (over 200M) to distribute as in the conventional FL, the extra variables to communicate are p̂’s, which only occupy about 16K storage. Thus, additional communication cost is negligible. Moreover, the total privacy cost is still of a low level with number M = 10 (i.e., communication rounds times the cost for each round).

heading: 5 CONCLUSIONS

text: With the carefully designed DPLC algorithm and a novel consensus-aware recognition loss, we improve federated learning performances on face recognition by communicating auxiliary embedding centers among clients, while achieving rigorous differential privacy. The framework runs efficiently with lightweight communication/computational overheads. Besides, PrivacyFace can be potentially extended to other metric-learning tasks such as re-identification and image retrieval. In the future, more efforts can be spent on designing a more accurate clustering algorithm in conjunction with the FL optimization, e.g., adaptive querying on the confusion areas instead of a brute-force sampling.

heading: 6 REPRODUCIBILITY STATEMENT

text:
Sec. 3.1 provides key proofs as well as properties of the proposed method. Additional proofs are described in Sec. A.1 of the appendix.
The involved training/test datasets and training configurations are detailed in Sec. 4 when using FedAvg scheme. Similar settings are applied for FedSGD in Sec. A.2.1 in appendix. Sec. A.2.4 of the appendix shows visualizations for two potential attacks: K-nearest neighbor attack and inversion attack. We as well present involved datasets, network structures, training losses as well as training schedules for these attacks. Those are sufficient for reproducibility.

heading: A APPENDIX

heading: A.1 MATHEMATICAL PROOFS

text: In this section, we present extra proofs for PrivacyFace.

heading: A.1.1 PROOF FOR THEOREM 1

text:
Proof. A spherical cap is a portion of a d-dimensional sphere with radius r cut off by a plane. As shown in (Li, 2011), the surface area of a spherical cap is
A = 1
2 Adr
d−1I(2rh−h2)/r2
( d− 1
2 ,
1
2
) .
Here h is the height of the cap, and A,Ad are the surface area of the spherical cap and the sphere, respectively. Because the set {f : arccos(fTp) ≤ ρ,f ∈ Sd} is the spherical cap with height h = r · (1− cos ρ) and the radius is 1 in our problem, it can be derived that the occupancy ratio is
A
Ad =
1 2 Adr d−1I(2r2·(1−cos ρ)−r2·(1−cos ρ)2)/r2
( d− 1
2 ,
1
2 ) = 1
2 Adr
d−1I(2− 2 cos ρ−1−cos2 ρ+ 2 cos ρ )
( d− 1
2 ,
1
2 ) = 1
2 Isin2(ρ)
( d− 1
2 ,
1
2
) .

heading: A.1.2 A WEAK VERSION OF THEOREM 2

text:
In this part, we present a straightfward but weak version of the Theorem 2: Theorem 3. Define a function p , f(S) = 1|S| ∑ i∈S wi. Then the Gaussian Mechanism p̂ ,
M(S) = 1|S| ∑ i∈S wi +N (0, σ2Id) is ( , δ)-DP if σ ≥ 4 |S|· √ (1− cos ρ) ln( 1.25δ ).
Proof. S stores indexes of class centers which have cosine similarities larger than cos ρwith a center (denoted as wo in this proof). Assuming that S,S ′ are neighbors differed at w,w′, then
‖f(S)− f(S ′)‖2 = 1 |S| ‖w −w ′‖2 ≤ 1 |S| (‖w −wo‖2 + ‖w ′ −wo‖2) ≤ 2 |S| √ 2− 2 cos ρ (6)
According to Definition 3, we conclude that M(S) = 1|S| ∑ i∈S wi + N (0, σ2Id) is ( , δ)-DP if
σ ≥ ∆2(f) √
2 ln( 1.25δ ) = 4 |S|· √ (1− cos ρ) ln( 1.25δ ).
This lower bound of σ is 4|S|· √ (1− cos ρ) ln( 1.25δ ), which is larger than our final bound because
( 4
|S| ·
√ (1− cos ρ) ln(1.25
δ ))2 − ( 2|S| ·
√ (1− cos(2ρ)) ln(1.25
δ ))2
= 4 |S|2 · 2 ln( 1.25 δ ) · (4− 4 cos ρ− 1 + cos(2ρ))
= 4 |S|2 · 2 ln( 1.25 δ ) · (3− 4 cos ρ+ 2 cos2 ρ− 1)
= 4 |S|2 · 2 ln( 1.25 δ ) · (2− 2 cos ρ)2 ≥ 0.
A tighter bound can reduce the privacy cost with same level of noise added. For example, the privacy cost in our experiments can be dropped by around 21% if using ρ = 1.3 (from = 1 to = 0.793).

heading: A.1.3 PROOF FOR THEOREM 1

text:
Lemma 1. For any x,y, z ∈ Rd, the following inequality holds:
(xTxzTz − xTzxTz) · (yTyzTz − yTzyTz) ≥ (xTzzTy − xTyzTz)2.
Proof. Let x = (x1, x2, · · · , xd),y = (y1, y2, · · · , yd), z = (z1, z2, · · · , zd). Then
xTxzTz − xTzxTz = d∑ i=1 x2i · d∑ j=1 z2j − ( d∑ i=1 xizi) 2
= d∑ i=1 d∑ j=1 x2i z 2 j − d∑ i=1 xizi · d∑ j=1 xjzj
= d∑ i=1 d∑ j=1 x2i z 2 j − d∑ i=1 d∑ j=1 xizixjzj = d∑ i=1 d∑ j=1 (x2i z 2 j − xizixjzj)
= 1
2 d∑ i=1 d∑ j=1 (x2i z 2 j + x 2 jz 2 i − 2xizixjzj)
= 1
2 d∑ i=1 d∑ j=1 (xizj − xjzi)2
Similarly, it’s easy to prove yTyzTz − yTzyTz = 12 ∑d i=1 ∑d j=1(yizj − yjzi)2. In addition, we have
xTzzTy − xTyzTz = d∑ i=1 xizi · d∑ j=1 zjyj − d∑ i=1 xiyi · d∑ j=1 zjzj
= d∑ i=1 d∑ j=1 (xizizjyj − xiyizjzj)
= 1
2 d∑ i=1 d∑ j=1 (xizizjyj − xiyizjzj + xjzjziyi − xjyjzizi)
= 1
2 d∑ i=1 d∑ j=1 (xizj · (ziyj − yizj) + xjzi · (zjyi − yjzi))
= 1
2 d∑ i=1 d∑ j=1 (xizj − xjzi)(ziyj − yizj)
Then we can deduce that
(xTxzTz − xTzxTz) · (yTyzTz − yTzyTz) = 1 2 d∑ i=1 d∑ j=1 (xizj − xjzi)2 1 2 d∑ i=1 d∑ j=1 (yizj − yjzi)2
≥ 1 4 ( d∑ i=1 d∑ j=1 (xizj − xjzi)(ziyj − yizj) )2 = (xTzzTy − xTyzTz)2
Here the second step is the Cauchy–Schwarz inequality.
Lemma 2. Assume thatw,w′,wo ∈ Rd and w T o w ‖wo‖2‖w‖2 = cosα, wTo w
′
‖wo‖2‖w′‖2 = cosβ, 0 ≤ α, β ≤ π 2 . Then wTw′ ‖w‖2‖w′‖2 ≥ cos(α+ β) always holds.
Proof. Let w Tw′
‖w‖2‖w′‖2 = cos γ. We thereby need to prove cos γ ≥ cos(α + β) = cosα cosβ − sinα sinβ = cosα cosβ − √ (1− cos2 α)(1− cos2 β), which is equivalent to√
(1− cos2 α)(1− cos2 β) ≥ cosα cosβ − cos γ. (7)
If cosα cosβ ≤ cos γ, the inequality equation 7 holds. If cosα cosβ > cos γ, then
(1− cos2 α)(1− cos2 β) ≥ (cosα cosβ − cos γ)2
⇐⇒ (1− ( w T o w ‖wo‖2‖w‖2 )2)(1− ( w T o w ′ ‖wo‖2‖w′‖2 )2) ≥ ( w T o w ‖wo‖2‖w‖2 wTo w ′ ‖wo‖2‖w′‖2 − w Tw′ ‖w‖2‖w′‖2 )2 ⇐⇒ ((‖wo‖2‖w‖2)2 − (wTo w)2)(‖wo‖2‖w′‖2)2 − (wTo w′)2) ≥ ((wTo w) · (wTo w′)−wTw′‖wo‖22)2 ⇐⇒ (wTwwTo wo −wTwowTwo) · (w′Tw′wTo wo −w′Twow′Two) ≥ (wTwowTo w′ −wTw′wTo wo)2.
The last step is correct according to Lemma 1.

heading: A.2 EXTRA EXPERIMENTS

heading: A.2.1 PRIVACYFACE WITH FEDSGD

text:
Algorithm 2 is mainly built on FedAvg (McMahan et al., 2017a) where model parameters are communicated between clients and the server. In this part, we show that our PrivacyFace is also compatible with another popular FL method called FedSGD (Shokri & Shmatikov, 2015). FedSGD mainly differs FedAvg in two aspects: (1) During training locally, the model is frozen to ensure gradients across mini-batches are from a same model. (2) Gradients rather than model parameters are aggregated and distributed among clients and the server. Apart from changing FedAvg to FedSGD, we let other training setting consistent with those described before and present the results in Tab. 2. Compared to the conventional FL method φ, performances of our PrivacyFace φ + p̂ are close to those on RFW and noteworthily higher in IJB-B/IJB-C benchmarks. Besides, φ + p̂ and φ + p achieve very close results. All these phenomenons are in accord with those observed when using FedAvg, which shows the scalability of the proposed PrivacyFace.
However, FedSGD does not achieve that significant improvements as FedAvg. A possible reason is that FedSGD naturally requires more communication rounds than FedAvg to achieve comparable results, which may be caused by low-frequent updates during training. Experiments in McMahan et al. (2017a) showed that FedSGD requires about 23 times more communication rounds to achieve similar performances with FedAvg. With such a fact, the improvements using FedSGD in PrivacyFace is still remarkable.

heading: A.2.2 COMPARISON WITH A NAIVE DP APPROACH

text:
A naive alternative to DPLC is to directly make each class center of W to be differential private by adding noise. However, that can lead to a large privacy cost due to two reasons:
1. With the same noise added, the privacy cost is propotional to the l2-sensitivity based on definition 2. The l2-sensitivity of the naive approach is 2 as maxinum distances between two class centers distributed in a unit sphere are 2. In contrast, the l2-sensitivity DPLC is 1 |S| √ 2− 2 cos(2ρ). In our experiment, we let |S| > 512 and ρ = 1.4, which makes the
l2-sensitivity to be smaller than 0.0024.
2. The training dataset are of a large number of identities. According to the composition rule (Definition 3), the privacy cost grows linearly with respect to the queries (i.e., number of identities). In DPLC, low privacy cost as the clustering mechanism highly reduce the number of queries.
During training, the number of query is 1. Therefore, the privacy cost in an local client (with 7000 classes) of DPLC is 1·0.0247000·2 ≈ 1.7e− 7 of that of the naive approach. We experimentally compare the performances of our method and the naive approach. For fair comparisons, the privacy cost in each round is fixed to be 1 and all training settings are consistent with those in the main text. Tab. 3 are the results, where the φ+ Ŵ represents the naive approach. It can be seen that φ+ Ŵ can achieve similar results on RFW benchmark as learning local discriminative features with introduced fixed class centers are relatively easy (for example, if introduced class center distributed in the south of the embedding space, placing local features to the north is a solution). However, as the class centers from other clients are very noisy, the global embedding space will become a mess. φ+ Ŵ even achieves much worse results than the conventional FL methods φ on the challenging benchmarks. The tar decreases 23.99% on IJB-B and 25.16% on IJB-C when far is 1e-4. In contrast, PrivacyFace increases the performances by 9.63% and 10.26% respectively, which shows the superiority of our framework.

heading: A.2.3 NECESSITY OF FEDERATED LEARNING

text:
Besides using federated learning, one can also choose to finetune client-expert models on local datasets. We also implement this approach by training four independent models on sub-datasets of BUPT-Balancedface and test their performances on benchmarks. The training procedure is consistent with what we described before in Sec. 4. Specifically, we finetune each model from φ0 by ArcFace for 10 epochs with learning rate 0.001, batch size 512 and weight decay 5e-4.
We present our results in Tab. 4, where the “African/Asian/Caucasian/Indian” in the first column means the model trained on the corresponding sub-dataset of BUPT-Balancedface. Models finetuned on African and Caucasian sub-dataset achieve satisfying results on local benchmarks while those on Asian and Indian have even worse performances than the initial model φ0. Moreover, all these local models perform poorly on the IJB-B and IJB-C benchmarks, which indicates the lack of generality. To conclude, local training highly relies on the properties and qualities of local datasets, and can not lead to general models. In contrast, models trained by conventional FL method and PrivacyFace can achieve competitive performances on all benchmarks, and therefore are more robust to various scenarios.
In this section, we experimentally examine the privacy attacks to demonstrate that class centers lead to privacy leakage while our sanitized clusters are resistent to these attacks. We mainly consider two types of attacks: K-nearest neighbor attack and reversion attack.
K-Nearest Neighbor Attack We assume that an attacker owns a gallery of faces with enormous identities and attacks the exposed class centers by finding their neighbors. To simulate the K-nearest neighbor attack, class centers are extracted on the BUPT-Balancedface dataset and three faces are sampled from each identity on a large dataset called MS1M-V2 (Deng et al., 2019). Fig. 6 presents the results. It can be observed that the K-nearest neighbor attack successfully find faces of the corresponding identity from the class centers, which enables the attacker to know which identities are contained in the training dataset. The observation further verifies the fact that sharing class centers leads to privacy leakage in face recognition.
Reversion Attack We design a reconstruction network (modified from DCGAN (Radford et al., 2015)) as described in Tab. 5 which projects a 512 dimensional embedding to a face image. CASIAWebFace dataset is used as the training dataset. Specifically, we extract features from images by the initial model φ0. With supervision of features and images, we train the network by Adam optimizer with L1 loss, which penalizes the absolute value of pixel differences between reconstructed and real images. The learning rate is initialized from 0.001 and divided by 10 every 10 epochs, and we stop the training at the 50th epoch. The weight decay is 5e− 4 and batch size is 512 during our training.
We test our model on BUPT-Balancedface dataset and present reconstruction visualizations. In each image pair of Fig. 7, the left one is the average of faces from one identity while the right one is the reconstructed image from the corresponding class center. It can be seen that reconstructed images reveal the real individual identity, which demonstrates thatw carries human face privacy and therefore should be protected. Fig. 8 shows the reconstructions from p and p̂ under various privacy cost . The left column is the images by p, which are much blurred than those from w and only the group properties can be identified. For example, the first image seems like an African male while
the second last one are very likely to be Caucasian female. The output p̂ from our DPLC algorithm
can further protect privacy as one can hardly recognize a face from the generated images, especially when is small. The hardness of a successful identification is negative related to according to the visualization.

heading: A.2.5 COMPARISONS WITH FEDFACE

text:
FedFace (Aggarwal et al., 2021) is a recent work applying federated learning to face recognition. In each communication round, the FedFace uploads all local class centers to the server and consequently utilizes the spreadout regularizer to separate class centers from different clients. In this section, we adjust FedFace to fit our federated setting in the main text. Specifically, in each communication round, each local client uploads its 7K class centers to the server. The spreadout regularizer then computes distances of 588 million (7K × 7K × 3 × 4) pairs of class centers and back propagates the model parameters once. The updated parameters are then distributed to each of the clients. However, directly computing all distances consumes 11,962M GPU memory, which is out of the capacity of our machine. Therefore, we sample 1K out of 7K class centers from each client for spreadout regularizer in our machine. For hyper-parameters of the spreadout regularizer, the authors recommend loss weight λ = 10 and margin v = 0.9. For fair comparisons, we further run 18 experiments with λ ∈ {0.5, 1, 1.5, 5, 10, 15} and v ∈ {0.3, 0.6, 0.9} and report the best results of FedFace.
Tab. 6 summarizes the overall comparisons between FedFace and PrivacyFace. We observe that
• Performance. FedFace only boosts the FedAvg by 0.82%, 2.27% on TAR@FAR=1e-4 for IJB-B and IJB-C. In contrast, PrivacyFace achieves 9.63%, 10.26%, respectively. Besides the sampling strategy, the main reason can be that FedFace only separates classes from different clients for one time (the spreadout regularizer) in a communication round, while the model is locally updated 0.3 million times (number of local images) without considering the conflicts among clients. In contrast, our consensus-aware recogntion loss takes the conflicts into account for each local update.
• Privacy cost. In each round, FedFace uploads 7, 000 original class centers to the server. Therefore, the privacy cost is 7000×∞ =∞. For PrivacyFace, the number is only 1 from the outputted noisy cluster center in the client.
• Additional communication cost. For each client in communication round, FedFace additionally uploads 7, 000 512-d class centers to the server and sends back the updated class centers. In PrivacyFace, a local client uploads one 512-d cluster centers to the server and downloads 4 cluster centers from the server. The additional cost from FedFace is 7000+7000 1+4 = 2, 800 times of that from PrivacyFace. • Additional computational cost. The additional computations in FedFace is from the
spreadout regularizer, which calculates 588 million pairs of 512-d features and updates the model parameters on the server side by one-time back propagation. That requires enormous computation in each round. In contrast, only extra items in the consensus-aware loss requires additional computation in PrivacyFace, which is of tiny volume.
• Additional usage of GPU memory. Even though we sample 1, 000 out of 7, 000 clients to reduce GPU usage of the spreadout regularizer, FedFace still consumes 1, 525M extra GPU memories. One the other hand, the additional usage of PrivacyFace is negligible, as shown in Tab. 6.

heading: A.2.6 ABLATION STUDIES ON INITIAL MODELS

text:
In this section, we examine the effects of different initial models on the proposed method. Besides the φ0 we used before, we further adopt two intermediate checkpoints at epoch 20, 10 when training on the CASIA-WebFace (Yi et al., 2014) and denote them as φ1,φ2, respectively. One additional
model φ3 is trained on MS1M-V2 (Deng et al., 2019) for 10 epochs with learning rate 0.1 and the ArcFace loss. Note that we do not follow the conventional training configuration where the learning rate is decreased stepwisely until 0.001. That’s because such a configuration will lead to a strong model that does not require finetuning on BUPT-BalancedFace (Wang & Deng, 2020).
With other training settings same as those in the main text, we finetune the models on BUPTBalancedFace (Wang & Deng, 2020) with the ArcFace loss. Tab. 7 presents the comparisons of plain FedAvg and PrivacyFace. On the IJB-B/IJB-C benchmarks, PrivacyFace can consistently achieve better results than the plain FedAvg. For TAR@FAR=1e-4 on IJB-C, PrivacyFace can boost the performances by 10.26%, 12.03%, 3.43%, 22.00% with the four initial models. It can be observed that improvements are affected by the initial model. For example, the poorest initial model φ2 achieves the lowest improvements. The reason may be that the training loss are overwhelmed by separating local classes as initial features are not discriminative enough. Consequently, increasing inter-client distances is of the secondary importance to the training in this scenario. For the best initial model φ3, the performances on IJB-B/C decreases when using federated learning. This is because the finetuning process also focuses on improving local performances as the global consensus is already reached. In this senario, FedAvg decreases the TAR@FAR=1e-4 by more than 40% on the benchmarks. In contrast, PrivacyFace can highly alleviate the performance degradations by taking global consensus into account during training. Finetuing on BUPT-BalancedFace mainly benefits the performances on the RFW benchmarks as these two datasets are both designed for racial bias. If the initial model is good enough, the finetuning procedure will focus more on fitting the racial issue instead of improving the general performances. This may be the reason why model φ3 leads to the largest improvements.

heading: A.2.7 EFFECTS OF CLIENT DISCONNECTIONS

text:
Previously we use a perfect federated setting where all clients are available during training. In realworld applications, some clients may drop for certain rounds of communication due to network issues. We further simulate the case by randomly select one of the four clients and exclude it from the training for an offline probability in each communication round. For example, if we set the offline probability to 100%, there will be exactly one client offline per round. Note here we only block one client because the total number of clients is small. We keep other settings unchanged and compare performances of PrivacyFace and plain FedAvg on the IJB-C benchmark with different offline probability. We refer the offline policy as disconnecting patterns during training, and use the same offline policy for each pair of experiments on PrivacyFace and FedAvg for fair purpose. To reduce the randomness, we run each experiment with 3 different offline policies and report the mean TAR@FAR=1e-4.
Fig. 9 presents our results. When the offline probability is 25% and 50%, PrivacyFace improves the performances of federated learning by about 6%. The improvement decreases to 2.4% with 75% offline probability. When each round witnesses one random client offline, the performances of two methods are below 45% while the improvement brought by our approach is only 0.4%. That reveals the performance boosts of PrivacyFace can be influenced by the network conditions.

heading: A.3 DIFFERENCES WITH DIFFERENTIAL PRIVATE FEDERATED LEARNING

text:
In PrivacyFace, we utilize the differential privacy to improve the performance of conventional federated learning in face recognition. Our work basically distills differentially private information from clients to achieve that purpose. We notice that there are plenty of recent works on differential private federated learning (DP-FL) methods (Geyer et al., 2017; McMahan et al., 2017b; Truex et al., 2019) and would like to emphasize the differences:
1. PrivacyFace and current DP-FL methods aim to solve different privacy leakages. As different data points lead to different model updates, DP-FL methods essentially add noise to model update to prevent data from being inspected. Recall the model update is essentially the average of gradients, which has a small l2-sensitivity (definition 2) thanks to the tremendous number of data points. Then the protection is relatively easy with the low privacy cost (propotional to the l2-sensitivity). In contrast, PrivacyFace aims to protects class centers in face recognition which are directly linked to individual privacy. The problem is much harder as these class centers are not naturally averaged like model updates and therefore leads to large privacy cost.
2. The targets to protect is different. Current DP-FL methods protect the data points in each local dataset. In contrast, PrivacyFace protects some of the model parameters (i.e., the classifier weights), which is the first work in federated learning to our best knowledge.
3. Current DP-FL methods theoretically decrease performances of federated learning as they use noisy updates. However, PrivacyFace distill differentially private information from each client. The extra information can broadcasted to improve the performances of federated learning face recognition.
4. We can directly combine DP-FL method and PrivacyFace, which lead to a more secure framework protecting both data points as well as class centers together.

title: IMPROVING FEDERATED LEARNING FACE RECOGNI-