  Learning from large amounts of unsupervised data and a small amount of
supervision is an important open problem in computer vision. We propose a new
semi-supervised learning method, Semantic Positives via Pseudo-Labels (SemPPL),
that combines labelled and unlabelled data to learn informative
representations. Our method extends self-supervised contrastive learning --
where representations are shaped by distinguishing whether two samples
represent the same underlying datum (positives) or not (negatives) -- with a
novel approach to selecting positives. To enrich the set of positives, we
leverage the few existing ground-truth labels to predict the missing ones
through a $k$-nearest neighbours classifier by using the learned embeddings of
the labelled data. We thus extend the set of positives with datapoints having
the same pseudo-label and call these semantic positives. We jointly learn the
representation and predict bootstrapped pseudo-labels. This creates a
reinforcing cycle. Strong initial representations enable better pseudo-label
predictions which then improve the selection of semantic positives and lead to
even better representations. SemPPL outperforms competing semi-supervised
methods setting new state-of-the-art performance of $68.5\%$ and $76\%$ top-$1$
accuracy when using a ResNet-$50$ and training on $1\%$ and $10\%$ of labels on
ImageNet, respectively. Furthermore, when using selective kernels, SemPPL
significantly outperforms previous state-of-the-art achieving $72.3\%$ and
$78.3\%$ top-$1$ accuracy on ImageNet with $1\%$ and $10\%$ labels,
respectively, which improves absolute $+7.8\%$ and $+6.2\%$ over previous work.
SemPPL also exhibits state-of-the-art performance over larger ResNet models as
well as strong robustness, out-of-distribution and transfer performance. We
release the checkpoints and the evaluation code at
https://github.com/deepmind/semppl .

Computer Vision and Pattern Recognition;Artificial Intelligence;Machine Learning