Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning
<|reference_start|>Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning: This paper delves into the interplay between vision backbones and optimizers, unvealing an inter-dependent phenomenon termed \textit{\textbf{b}ackbone-\textbf{o}ptimizer \textbf{c}oupling \textbf{b}ias} (BOCB). We observe that canonical CNNs, such as VGG and ResNet, exhibit a marked co-dependency with SGD families, while recent architectures like ViTs and ConvNeXt share a tight coupling with the adaptive learning rate ones. We further show that BOCB can be introduced by both optimizers and certain backbone designs and may significantly impact the pre-training and downstream fine-tuning of vision models. Through in-depth empirical analysis, we summarize takeaways on recommended optimizers and insights into robust vision backbone architectures. We hope this work can inspire the community to question long-held assumptions on backbones and optimizers, stimulate further explorations, and thereby contribute to more robust vision systems. The source code and models are publicly available at https://bocb-ai.github.io/.<|reference_end|>
@article{li2024unveiling,
    title={Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation
  Learning},
    author={Siyuan Li, Juanxi Tian, Zedong Wang, Luyuan Zhang, Zicheng Liu,
  Weiyang Jin, Yang Liu, Baigui Sun, Stan Z. Li},
    journal={arXiv preprint arXiv:2410.06373},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.06373},
    primaryClass={cs.CV cs.LG}
}