"Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize races-even races that are not represented in the training data-and identities.
We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be "re-purposed" for a different, privacy-violating task even in the absence of the original training data.
We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.
This paper introduces the problem of overlearning, which can be thought of as unintended transfer learning from a (victim) source model to a target task that the source model’s creator had not intended its model to be used for. The paper raises good points about privacy legislation limitations due to the fact that overlearning makes it impossible to foresee future uses of a given dataset.
Background on censoring is well developed, and helps position the submission. However, the relation to transfer learning is not sufficiently outlined in the introduction. In some ways, overlearning is a form of unintended transfer learning. In particular, the connection appears explicit in the case of model repurposing (Section 3.2). Editing the introduction to tease apart this relationship to transfer learning would help readers forge an intuition for what the paper considers.
While the de-censoring algorithm is intuitive, it is not clear what assumptions are being made because of ambiguous notation in Algorithm 1. How does Algorithm 1 train E_aux and C_aux? Does that step assume the adversary has knowledge of the algorithm use to train E and C? Descriptions of the experimental setup from Section 4.2 seem to indicate that this is the case. This is not necessarily an issue if the proposed attack is demonstrating a limitation of learning rather than a practical attack. 
Experiments overall show that censoring does not mitigate overlearning in a way that is robust to de-censoring. One aspect of the experiments that is unclear is the auxiliary dataset: what is the auxiliary dataset used by the adversary for each of the datasets considered in experiments? 
Question: how does simultaneous censoring of all layers affect overlearning and repurposing?
Question: what is the connection between learning more complex representations and overlearning? Is the intuition that if the representation is more complex, it is more likely to contain features useful to identify the sensitive attribute?
The paper is overall well-written. Some parts of the paper omit important details (perhaps due to space constraints?), but an editorial pass should address most of these. Detailed feedback:
1 / Explaining what model partitioning means in this context would help make the introduction more self-contained. 
1 / Is overlearning specific to attributes that raise fairness issues? Or is this phenomenon more general? The “Censoring representations” paragraph on page 2 seems to indicate that the phenomenon is more general.
5/ Is accuracy the best metric to report (e.g., for race attribute prediction) given that the random guessing figure suggests the data is not balanced across attribute values?
6/ Why is the effect not monotonic in Table 4? Were multiple runs averaged?