title: Decoupled Contrastive Learning
abstract: Contrastive learning (CL) is one of the most successful paradigms for self-supervised learning (SSL). Specifically, contrastive learning treats two augmented ``views'' of the same sample as positive, pulling them close and treating all other samples as negative to push them far apart. Despite the evident success of CL SSL methods, there are several challenges in the existing methods as they may require special structures, large batches, or huge training epochs, etc. Our motivation in this work is to provide a simple, efficient, and yet competitive contrastive learning baseline. Through both theoretical and empirical studies, we identified a strong negative-positive-coupling (NPC) effect in the widely used cross-entropy loss in CL SSL methods. We hypothesize that the NPC effect may be a major cause of the inefficiency in many contrastive learning methods. By removing the NPC effect, we reach a decoupled contrastive learning (DCL) objective function, which significantly improves the training efficiency. DCL can achieve competitive performance, requiring neither large batches in SimCLR, momentum encoding in Moco, or large epochs. We demonstrate the benefit of DCL in various benchmarks. Further, DCL is also much less sensitive to suboptimal hyperparameters. Notably, our approach achieves $66.9\%$ ImageNet top-1 accuracy with 256 batch size within 200 epochs pre-training, which outperforms its baseline SimCLR by $5.1\%$. We believe DCL may provide a strong baseline for future contrastive learning-based SSL studies.
authors: Chun-Hsiao Yeh Cheng-Yao Hong Yen-Chi Hsu Tyng-Luh Liu Yubei Chen Yann LeCun
hIndex: Yann LeCun 120