Understanding the Benefits of SimCLR Pre-Training in Two-Layer Convolutional Neural Networks
<|reference_start|>Understanding the Benefits of SimCLR Pre-Training in Two-Layer Convolutional Neural Networks: SimCLR is one of the most popular contrastive learning methods for vision tasks. It pre-trains deep neural networks based on a large amount of unlabeled data by teaching the model to distinguish between positive and negative pairs of augmented images. It is believed that SimCLR can pre-train a deep neural network to learn efficient representations that can lead to a better performance of future supervised fine-tuning. Despite its effectiveness, our theoretical understanding of the underlying mechanisms of SimCLR is still limited. In this paper, we theoretically introduce a case study of the SimCLR method. Specifically, we consider training a two-layer convolutional neural network (CNN) to learn a toy image data model. We show that, under certain conditions on the number of labeled data, SimCLR pre-training combined with supervised fine-tuning achieves almost optimal test loss. Notably, the label complexity for SimCLR pre-training is far less demanding compared to direct training on supervised data. Our analysis sheds light on the benefits of SimCLR in learning with fewer labels.<|reference_end|>
@article{zhang2024understanding,
    title={Understanding the Benefits of SimCLR Pre-Training in Two-Layer
  Convolutional Neural Networks},
    author={Han Zhang and Yuan Cao},
    journal={arXiv preprint arXiv:2409.18685},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.18685},
    primaryClass={cs.LG stat.ML}
}