We present a novel approach G-PATE for training a scalable differentially private data generator, which can be used to produce synthetic datasets with strong privacy guarantee while preserving high data utility. Our approach leverages generative adversarial nets to generate data and exploits the PATE (Private Aggregation of Teacher Ensembles) framework to protect data privacy. Compared to existing methods, our approach significantly improves the use of privacy budget. This is possible since we only need to ensure differential privacy for the generator, which is the part of the model that actually needs to be published for private data generation. In particular, we connect a student generator with an ensemble of teacher discriminators and propose a private gradient aggregation mechanism to ensure differential privacy on all the information that flows from the teacher discriminators to the student generator. Theoretically, we prove that our algorithm ensures differential privacy for the generator. Empirically, we provide thorough experiments to demonstrate the superiority of our method over prior work on both image and non-image datasets.
This paper studies the problem of differentially private data generator. Inspired by the general GAN framework and the PATE mechanism, the authors propose a new differentially private training algorithm for data generator. The problem of training data generator with privacy guarantee considered in this paper is very interesting, and the proposed algorithm looks novel. However, there are lots of unclear statements in the current paper, and I cannot tell whether the proposed algorithm is indeed better than previous methods. Following are my major concerns:
1.It is unclear what are the loss functions used in equation (1) and (2). Please define k when introducing equation (2).
2.The training framework introduced in section 3.1 is different from the traditional GAN framework, and thus my concern is that whether this framework will give us good generated samples. Because the performance of GAN has been proved in both theory and practice. The authors should at least empirically show the performance of the proposed framework in the nonprivate setting.
3.There is no introduction of the $(\epsilon,\delta)$-differential privacy before introducing the Definition 1.
4.There is no definition of Renyi differential privacy, so the statement of Theorem 2 is unclear. In addition, what is data-dependent Renyi differential privacy?
5.The privacy guarantee of Algorithm 2 is not very clear. Because there are lots of parameters in Algorithm 1 and 2 which may affect the privacy guarantee, and Theorem 3 does not state such requirements. For example, how to choose $\sigma_1,\sigma_2$? In Theorem 7, there are some constraints on different parameters, will them be satisfied by your algorithm?
6.How will the number of teacher models affect the privacy guarantee?
7.Why you choose random projection matrix with variance $1/k$, and what is the projection dimension for different algorithms?
8.In Table 1, the results of non private GAN are different from the results of non private GAN reported in PATE-GAN paper. Since the baseline results are much better in the current than the results reported in the PATE-GAN paper, it seems to me that the improvements of the proposed method comes from the stronger baseline.
Other comments:
1.$\lambda>1$ in Theorem 3.
2.Algorithm 2 should be moved to main context.
3.The last sentence in section 3.2 is not convincing.
4.Typo “differnet” in the caption of Table 1.