Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate overfitting. This paper considers the related question of "membership inference", where the goal is to determine if an image was used during training. We consider membership tests over either ensembles of samples or individual samples.
First, we show how to detect if a dataset was used to train a model, and in particular whether some validation images were used at train time. Then, we introduce a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples. To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.
==============Final Evaluation================
I have gone through the other reviews as well as the author response.
Firstly, I would like to thank the authors for providing detailed responses to my questions.
In general, I agree with R2 that the paper generally has some potentially interesting ideas and results but the manner in which the current draft is organized and presented makes it hard to grasp them and there is a lack of coherent message about what the paper is about.
Moreover, from my understanding the analysis in David McKay’s book (Chapter 41) concerns a single neuron (and the number of parameters for a single neuron). As pointed out by R2, with depth there are a lot more number of possible ways in which one could carve out decision boundaries to separate data points, thus, it is not clear that the loose linear upper bound holds Specifically, as one might expect with depth it could be possible that linear capacity increase is a lower bound (I am not suggesting that it is, but that possibility should be considered and explained in the paper). Similarly, it would be good to formally connect the capacity to the rate of memorization before making a statement about them being related (as suggested in the initial review). In general, I feel this section could use some tighter formalism and justifications.
I also remain unconvinced by the response to my issue with the claim “Our experiments show that our networks can remember a large number of images and distinguish them from unseen images”, where the negative images are also seen by the memorization model, so they are not unseen. The authors address this by saying 3M of the 15 M negatives have been seen. That does not seem like a small enough percentage to claim that these are “unseen” images.
In general, I feel the paper is interesting but would benefit from a major revision which makes the message of the paper more clear, and addresses these and other issues raised in the review phase. Thus I am holding my current rating.
==================
Summary
The paper trains classification models to classify a labeling of a subset of images (assigned with label 1) from the rest of the images (assigned with a label 0). Firstly, the paper shows that deep learning models are able to learn such classifiers and get low training loss. It then proposes to use this model to ``attack’’ task-specific models to perform membership inference, i.e. figuring out if an image provided in a set was used in training or not. 
Strengths
+ The paper thoroughly covers related work and provides context.
+ Results on confidence as a signature of a dataset are interesting.
Weaknesses
[Motivation]
1. In general, recent work has found that the raw number of parameters has little to do with the size of the model class or the capacity of a model for deep models, and thus work like [A] has been trying to come up with better complexity measures for models to explain generalization. Thus, without sufficient justification the assertion in the paper that the capacity of the network is well approximated by the number of parameters does not seem correct. Also, the claim in Fig. 1 that the transition from ‘’high capacity’’ to low capacity happens at the number of parameters in the network seems a bit loose and hard to substantiate from what I understand, and should be toned down. (*)
[Capacity]
2. Sec. 3.3, Fig. 3: The capacity (in terms of parameters)of both Resnet-18 and VGG-16 is higher than the capcity for YFCC100M dataset for n=10K images (comes to 161K bits), while the capacity of Resnet-18, with 14.7 million parameters (assuming float32 encoding) has 14.7 * 32 bits = 470.4 million bits, thus capacity alone cannot explain why VGG converges faster than Resnet-18, since both networks exceed the capacity, and capacity does not seem to have an established formal connection to rate of memorization. This is something which would need to be explained/ substantiated separately. (*)
3. Scenario discussed in Sec. 4 seems somewhat impractical. Given a set of m images, it is not clear that a classifier that is trained to detect between train and validation is sufficient, as one might also need to figure out if it is neither train nor val, which is a very practical scenario.
4. Fig. 3 (right): It is not clear why the fact that the classifier is able to predict which dataset the image ‘m’ corresponds to is useful or practical, as this seems to be a property of the set ‘m’ rather than the property of the trained classification model (f_\theta). Please clarify. On the other hand it is clear that using the confidence of the model to predict the dataset is a useful property, but the right side of the Fig. is very confusing. (*)
6. It is not clear to me what the point of Sec. 5 is, given a trained model, one wants to figure out if an image was present in the training of the model. While the baseline approaches seem to make use of the model confidence, I cannot see how the proposed approach (which uses a classifier) makes use of the original model. It is also not clear why Table. 3 does not report the Bayes baseline results. Also, does this section use the classifier for predicting the dataset, or is the approach reported in the section, the MAT approach?
7. ``Our experiments show that our networks can remember a large number of images and distinguish them from unseen images’’ -- this does not seem to be true, since the model is trained on both n as well as N -n ``unseen’’ images which it labels as the negative class, thus the negative class is also seen by the memorization model. (*)
Minor Points
1. It is not clear that training a network to classify a set from another set is necessarily equivalent to ``memorization’’. In addition, the paper would also need to show that such a model does not generalize to a validation set of images. This is probably obvious given the results from Zhang et.al. but should be included as a sanity check.
2. Figure 3: it is confusing to call the cumulative distribution of the maximum classification score as the CDF of the model (y-axis fig. 3 left) as CDF means something else generally in such contexts, as the CDF of a predictor.
References:
[A]: Blier, Léonard, and Yann Ollivier. 2018. ``The Description Length of Deep Learning Models.’’ arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1802.07044.
Preliminary Evaluation
There are numerous issues with the writing and clarity of the paper, while it seems like some of the observations around the confidence of classifiers are interesting, in general the connection between those set of results and the ``memorization’’ capabilities of the classifier trained to remember train vs val images is not clear in general. Important points for the rebuttal are marked with (*).