It is clear that users should own and control their data and privacy. Utility providers are also becoming more interested in guaranteeing data privacy. Therefore, users and providers can and should collaborate in privacy protecting challenges, and this paper addresses this new paradigm. We propose a framework where the user controls what characteristics of the data they want to share (utility)   and what they want to keep private (secret), without necessarily asking the utility provider to change its existing machine learning algorithms. We first analyze the space of privacy-preserving representations and derive natural informationtheoretic bounds on the utility-privacy trade-off when disclosing a sanitized version of the data X. We present explicit learning architectures to learn privacypreserving representations that approach this bound in a data-driven fashion. We describe important use-case scenarios where the utility providers are willing to collaborate with the sanitization process. We study space-preserving transformations where the utility provider can use the same algorithm on original and sanitized data, a critical and novel attribute to help service providers accommodate varying privacy requirements with a single set of utility algorithms. We illustrate this framework through the implementation of three use cases; subject-withinsubject, where we tackle the problem of having a face identity detector that works only on a consenting subset of users, an important application, for example, for mobile devices activated by face recognition; gender-and-subject, where we preserve facial verification while hiding the gender attribute for users who choose to do so; and emotion-and-gender, where we hide independent variables, as is the case of hiding gender while preserving emotion detection.
will make such devices not to understand the data until the visual or sound trigger is detected, with the capability to do this at the sensor level and without modifying the existing recognition system. This new paradigm of collaborative privacy environment is critical since it has also been shown that algorithmic or data augmentation and unpredictable correlations can break privacy Israel et al. (2014); Narayanan & Shmatikov (2008); Oh et al. (2016); Reuben et al. (2016). The impossibility of universal privacy protection has been studied extensively in the domain of differential privacy Dwork (2008), where a number of authors have shown that assumptions about the data or the adversary must be made in order to be able to provide utility Dwork & Naor (2010); Hardt et al. (2016); Kifer & Machanavajjhala (2011;. We can, however, minimize the amount of privacy we are willing to sacrifice for a given level of utility. Other recent data-driven privacy approaches like Wu et al. (2018) have also explored this notion, but do not integrate the additional collaborative constraints.
Therefore, it is important to design collaborative systems where each user shares a sanitized version of their data with the service provider in such a way that user-defined non-sensitive tasks can be performed but user-defined sensitive ones cannot, without the service provider requiring to change any data processing pipeline otherwise.
Contributions-We consider a scenario where a user wants to share a sanitized representation of data X in a way that a latent variable U can be inferred, but a sensitive latent variable S remains hidden. We formalize this notion using privacy and transparency definitions. We derive an informationtheoretic bound on privacy-preserving representations. The metrics induced by this bound are used to learn such a representation directly from data, without prior knowledge of the joint distribution of the observed data X and the latent variables U and S. This process can accommodate for several user-specific privacy requirements, and can be modified to incorporate constraints about the service provider's existing utility inference algorithms enabling several privacy constraints to be satisfied in parallel for a given utility task.
We apply this framework to challenging use cases such as hiding gender information from a facial image (a relatively easy task) while preserving subject verification (a much harder task), or designing a sanitization function that preserves subject identification on a consenting subset of users, while disallowing it on the general population. Blocking a simpler task while preserving a harder one and blocking a device from constantly listening out-of-sample data are new applications in this work, here addressed with theoretical foundations and respecting the provider's existing algorithms, which can simultaneously handle sanitized and non-sanitized data.
The problem statement is detailed in Section 2, and the information-theoretic bounds are derived in Section 3. Section 4 defines a trainable adversarial game that directly attempts to achieve this bound; the section also discusses how service-provider specific requirements can be incorporated. Examples of this framework are shown in Section 5. The paper is concluded in Section 6. Complementary information and proofs are presented in the Supplementary Material .
[Second update] I'd like to thank the authors for their detailed response. The authors have made changes that I believe improve the overall quality of the submission. I now lean towards accepting the paper, and have increased my rating from a 5 to a 6.
Most notably: (i) they clarified that their secret-detection model was retrained on sanitized data in their experiments, (ii) they added details about their experimental setup and the algorithms used for their experimental evaluation, and (iii) they added experiments to the appendix of the submission that evaluated their framework on synthetic data. I do, however, still have some concerns about how well the privacy guarantees of the proposed algorithm would hold up in practice against a motivated adversary (since formal privacy guarantees appear to be relatively weak right now).
As a minor comment, there may be a typo in Equation 20 of Section 7.2: the case (u, s) = (1, 0) is handled twice, whereas the case (u, s) = (0, 0) is never handled at all.
[First update] I find the authors' problem statement appealing, but share concerns with Reviewer 1 about the privacy guarantees offered by the proposed method, and with Reviewer 3 about need to clarify the experimental evaluation. No author response was provided; I've left my score for the paper unchanged. (Note: this update was posted a few days before the end of the rebuttal period; the submission was subsequently updated.)
[Summary] The authors consider a problem related to de-identification, where the goal is to perturb a dataset X in a way that makes it possible to infer some useful information U about each example in the dataset while obscuring some sensitive information S. For example, the authors consider the problem of perturbing pictures of people's faces to obfuscate the subjects' emotions while making it possible to infer their genders. The concrete approach explored in the paper's experimental evaluation ensures that an existing model trained model on the original dataset will continue to work when applied to the perturbed data.
On the theory side, the authors derive information-theoretic lower bounds on the extent to which one can disclose useful information about a dataset without leaking sensitive information, and propose concrete minimization problems that can perturb the data to trade off between the two objectives. On the practical side, the authors evaluate the minimization setup on three different problems.
[Key Comments] I'm of two minds about this paper. On the whole, I found the problem statement compelling. However, I had serious reservations about the implementation. First: I had trouble understanding the experimental setup based on the limited information provided in Section 5, and the results seem difficult to reproduce from the information in the paper. Second and more seriously: the security guarantees provided in practice seem very weak. At the very least, the authors should check whether their perturbations are robust against an adversary who retrains their model from scratch on perturbed data. This experiment would significantly strengthen the submission, but would still leave open the possibility that a clever adversary could extract more sensitive information than expected from the perturbed data.
[Details]
[Pro #1] The idea of perturbing an input in order to optimize bounds on how much "useful" versus "secret" information is disclosed by the output seems intuitively appealing. In that context, the theory from Sections 2 and 3 seems well-motivated. Section 3.2 ("defining a trainable loss metric") is especially well-motivated. It provides a concrete objective function which, when minimized, can obfuscate data in a way that trades off between utility and secrecy.
[Pro #2] The idea of perturbing a dataset in a way that allows existing useful algorithms to continue working without modifications seems like an interesting and novel contribution. I found the following excerpt from the introduction especially compelling: "it is important to design collaborative systems where each user shares a sanitized version of their data with the service provider in such a way that user-defined non-sensitive tasks can be performed but user-defined sensitive ones cannot, without the service provider requiring to change any data processing pipeline otherwise."
[Pro #3] The paper combines theoretical results with empirical case studies on three different problems. Based on visual inspection, the outputs of the perturbation heuristics shown in Section 5 / Figure 3 and Figure 4 seem reasonable.
[Con #1] Few details are provided about the experimental setup used in Section 5, and it was difficult for me to understand how the theoretical results in Section 4 were actually being applied. There's typically a lot of work that goes into turning a theoretical objective function (e.g., Equation 10 in Section 4.2) into a practical experimental setup. This could be a major contribution of the paper. But right now, I feel like there aren't enough details about the implementation for me to reproduce the experiments.
[Con #2] I had trouble understanding the motivation for the Subject within Subject case study in Section 5.1. The authors describe the problem as follows: "Imagine a subset of users wish to unlock their phone using facial identification, while others opt instead to verify their right to access the phone using other methods; in this setting, we would wish the face identification service to work only on the consenting subset of users, but to respect the privacy of the remaining users." The proposed solution (Figure 3) applies minor perturbations to the pictures of consenting subjects while editing the photos of the non-consenting users to leave only their silhouettes. A simple baseline would be to remove the photos of the non-consenting users from the dataset entirely. The case study would greatly benefit from a discussion of why the baseline is insufficient. It's also perfectly reasonable to say that the section is meant as a way to check whether the objective function from Section 4 can lead to reasonable behavior in practice, but if so, the intent should be clarified.
[Con #3] As far as I can tell, the practical experiments in Section 5 assume that the party who perturbs the dataset knows exactly what algorithm an attacker will use to infer secret information. They also seem to assume that the attacker cannot switch to a different algorithm -- or even retrain an existing machine-learned model -- to try and counter the perturbation heuristics. From the beginning of Section 5: "Initially, we assume that the secret algorithm is not specifically tailored to attack the proposed privatization, but instead is a robust commonly used algorithm trained on raw data to infer the secret." Unless I missed something, it seems like this assumption is used throughout the experimental evaluation.
To the authors' credit, the submission states this assumption explicitly in Section 5. From a security perspective, however, this seems like a dangerous assumption to rely on, as it leaves "sanitized" data vulnerable to attacks. For example, an attacker with knowledge of the perturbation algorithm can retrain the model they use to extract sensitive information, using perturbed images in place of the original images in their training dataset.
My main practical concern is that the security guarantees provided by the submission seem fragile. It may be much easier to build a perturbation algorithm that is resistant to a single (known) attack than to remove the sensitive information from the dataset entirely. Right now, the empirical results in the submission seem to focus on the former.