The lottery ticket hypothesis argues that neural networks contain sparse subnetworks, which, if appropriately initialized (the winning tickets), are capable of matching the accuracy of the full network when trained in isolation. Empirically made in different contexts, such an observation opens interesting questions about the dynamics of neural network optimization and the importance of their initializations. However, the properties of winning tickets are not well understood, especially the importance of supervision in the generating process. In this paper, we aim to answer the following open questions: can we find winning tickets with few data samples or few labels? can we even obtain "good" tickets without supervision? We find that winning tickets found in these scenarios are, perhaps surprisingly, competitive with winning tickets generated on the full ImageNet dataset when evaluated on ImageNet classification task.
This paper empirically studies the lottery ticket hypothesis with limited or no supervision. First, the authors use self-supervised learning to generate winning tickets, showing that "good" (reasonable) winning tickets can be found without labels. Second, the authors show that finding "good" (reasonable) winning tickets can be accelerated by a factor 5 on ImageNet by using only a subset of the data. The authors also argue that using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading.
The experimental results are rich and provide more understanding of winning ticket generation with limited or no supervision. The results on self-supervised learning task (including the layer-wise pruning results) and a subset of training dataset are reasonable and kind of expected, but it is still good that this paper provide solid experimental results to verify this. As the paper observed, "none of the tickets found with limited access to labels and or data matches the accuracy of tickets found with all the labeled data when considering moderate pruning rates (more than 10% of unpruned weights)
on ImageNet. Indeed, we consistently observe a decrease in performance compared to the full overparametrized network as soon as we prune the network." In this sense, winning tickets are certainly label and data dependant. This undermines the *bold* claim in the abstract that "we provide a positive answer to both questions, by generating winning tickets with limited access to data, or with self-supervision". From my perspective, the ability to exactly perserve the accuracy while pruning the weights (see the flat regions of "Lables" curves in Figure 1,2,3,4,5) is the interesting part of the lottery ticket hypothesis. We have several different ways to achieve a descreased accuracy with a smaller network, the dynamics there may be a mixture of the lottery ticket hypothesis and standard model pruning, which needs more careful experiment design to separate different dynamics.
"using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading." "The definition of “early in training” is somehow ill-defined: network
weights change much more for the first epochs than for the last ones." These two messages are important to future study of the lottery ticket hypothesis. This paper raises the issue of ill-definedness of “early in training”, but did not provide a solution. 
Overall, I found that the experimental results in this paper are solid and provide more understandings of the lottery ticket hypothesis. However, I feel that the novelty of this paper is limited, and do not provide much new insights. Therefore, it does not reach the bar of being published at ICLR, from my perspective. Therefore, I say "Weak Reject".