title: Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation
    Learning
summary: This paper delves into the interplay between vision backbones and optimizers,
  unvealing an inter-dependent phenomenon termed
  \textbf{backbone-optimizer coupling bias}
  (BOCB). We observe that canonical CNNs, such as VGG and ResNet, exhibit a
  marked co-dependency with SGD families, while recent architectures like ViTs
  and ConvNeXt share a tight coupling with the adaptive learning rate ones. We
  further show that BOCB can be introduced by both optimizers and certain
  backbone designs and may significantly impact the pre-training and downstream
  fine-tuning of vision models. Through in-depth empirical analysis, we summarize
  takeaways on recommended optimizers and insights into robust vision backbone
  architectures. We hope this work can inspire the community to question
  long-held assumptions on backbones and optimizers, stimulate further
  explorations, and thereby contribute to more robust vision systems. The source
  code and models are publicly available at https://bocb-ai.github.io/.
paper_authors._id: 67073a7a95f90bdb8e86a0ff
paper_authors.name: Siyuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:18:06.834Z
paper_authors._id: 640f7083208821a59b74c757
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678735253848-640f7083208821a59b74c757.jpeg
paper_authors.fullname: Siyuan Li
paper_authors.type: user
paper_authors.user: Lupin1998
paper_authors._id: 67073a7a95f90bdb8e86a100
paper_authors.name: Juanxi Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:44:18.996Z
paper_authors._id: 670880950e79a8b46f7ff9dd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/670880950e79a8b46f7ff9dd/hA1TLhwlQblkFsq8wLrkB.jpeg
paper_authors.fullname: Juanxi Tian
paper_authors.type: user
paper_authors.user: Juanxi
paper_authors._id: 67073a7a95f90bdb8e86a101
paper_authors.name: Zedong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:01:29.377Z
paper_authors._id: 6594d390674349122ce6f368
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6594d390674349122ce6f368/luDBiSMX_9l8QEpAQu3HJ.jpeg
paper_authors.fullname: Zedong Wang
paper_authors.type: user
paper_authors.user: ZedongWangAI
paper_authors._id: 67073a7a95f90bdb8e86a102
paper_authors.name: Luyuan Zhang
paper_authors._id: 67073a7a95f90bdb8e86a103
paper_authors.name: Zicheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:26:46.409Z
paper_authors._id: 6569f6cb44ce94a70187f407
paper_authors.avatarUrl: /avatars/414ecc9739ed23f9f395bd7cfa65055f.svg
paper_authors.fullname: Zicheng Liu
paper_authors.type: user
paper_authors.user: ZCLiu35
paper_authors._id: 67073a7a95f90bdb8e86a104
paper_authors.name: Weiyang Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:26:53.527Z
paper_authors._id: 66608add236f958513d21d2e
paper_authors.avatarUrl: /avatars/53eca0891c98cbb93be899885160a983.svg
paper_authors.fullname: Weiyang Jin
paper_authors.type: user
paper_authors.user: SOTA-Owner
paper_authors._id: 67073a7a95f90bdb8e86a105
paper_authors.name: Yang Liu
paper_authors._id: 67073a7a95f90bdb8e86a106
paper_authors.name: Baigui Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:27:02.990Z
paper_authors._id: 64d9f1409a6a7ae9846ab8d1
paper_authors.avatarUrl: /avatars/208b9d05b30f0094c3301936fabf4656.svg
paper_authors.fullname: Baigui Sun
paper_authors.type: user
paper_authors.user: sunbaigui
paper_authors._id: 67073a7a95f90bdb8e86a107
paper_authors.name: Stan Z. Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:27:08.710Z
paper_authors._id: 64d187bf5de9e1e911d4777f
paper_authors.avatarUrl: /avatars/0290d06655340ea57ec3db0b5f89455f.svg
paper_authors.fullname: Stan Z. Li
paper_authors.type: user
paper_authors.user: szli-0000
paper_title: Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation
    Learning
paper_summary: This paper delves into the interplay between vision backbones and optimizers,
  unvealing an inter-dependent phenomenon termed
  \textbf{backbone-optimizer coupling bias}
  (BOCB). We observe that canonical CNNs, such as VGG and ResNet, exhibit a
  marked co-dependency with SGD families, while recent architectures like ViTs
  and ConvNeXt share a tight coupling with the adaptive learning rate ones. We
  further show that BOCB can be introduced by both optimizers and certain
  backbone designs and may significantly impact the pre-training and downstream
  fine-tuning of vision models. Through in-depth empirical analysis, we summarize
  takeaways on recommended optimizers and insights into robust vision backbone
  architectures. We hope this work can inspire the community to question
  long-held assumptions on backbones and optimizers, stimulate further
  explorations, and thereby contribute to more robust vision systems. The source
  code and models are publicly available at https://bocb-ai.github.io/.
paper_ai_keywords: backbone-optimizer coupling bias CNNs VGG ResNet SGD families ViTs ConvNeXt adaptive learning rate optimizers pre-training downstream fine-tuning vision models robust vision backbone architectures