title: The Power of Contrast for Feature Learning: A Theoretical Analysis
abstract: Contrastive learning has achieved state-of-the-art performance in various self-supervised learning tasks and even outperforms its supervised counterpart. Despite its empirical success, the theoretical understanding of why contrastive learning works is still limited. In this paper, (i) we provably show that contrastive learning outperforms autoencoder, a classical unsupervised learning method, on both feature recovery and downstream tasks; (ii) we also illustrate the role of labeled data in supervised contrastive learning. This provides theoretical support for recent findings that contrastive learning with labels improves the performance of learned representations in the in-domain downstream task,  but it can harm the performance in transfer learning. We verify our theory with numerical experiments.
authors: Wenlong Ji Zhun Deng Ryumei Nakada James Zou Linjun Zhang
hIndex: James Zou 26