Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation
  Learning
This paper delves into the interplay between vision backbones and optimizers,
unvealing an inter-dependent phenomenon termed
\textbf{backbone-optimizer coupling bias}
(BOCB). We observe that canonical CNNs, such as VGG and ResNet, exhibit a
marked co-dependency with SGD families, while recent architectures like ViTs
and ConvNeXt share a tight coupling with the adaptive learning rate ones. We
further show that BOCB can be introduced by both optimizers and certain
backbone designs and may significantly impact the pre-training and downstream
fine-tuning of vision models. Through in-depth empirical analysis, we summarize
takeaways on recommended optimizers and insights into robust vision backbone
architectures. We hope this work can inspire the community to question
long-held assumptions on backbones and optimizers, stimulate further
explorations, and thereby contribute to more robust vision systems. The source
code and models are publicly available at https://bocb-ai.github.io/.
67073a7a95f90bdb8e86a0ff
Siyuan Li
claimed_verified
2024-10-13T20:18:06.834Z
640f7083208821a59b74c757
https://cdn-avatars.huggingface.co/v1/production/uploads/1678735253848-640f7083208821a59b74c757.jpeg
Siyuan Li
user
Lupin1998
67073a7a95f90bdb8e86a100
Juanxi Tian
claimed_verified
2024-10-11T07:44:18.996Z
670880950e79a8b46f7ff9dd
https://cdn-avatars.huggingface.co/v1/production/uploads/670880950e79a8b46f7ff9dd/hA1TLhwlQblkFsq8wLrkB.jpeg
Juanxi Tian
user
Juanxi
67073a7a95f90bdb8e86a101
Zedong Wang
claimed_verified
2024-10-10T08:01:29.377Z
6594d390674349122ce6f368
https://cdn-avatars.huggingface.co/v1/production/uploads/6594d390674349122ce6f368/KdWz6lZyGYQpjAgBDeiC1.jpeg
Zedong Wang (Jacky)
user
ZedongWangAI
67073a7a95f90bdb8e86a102
Luyuan Zhang
67073a7a95f90bdb8e86a103
Zicheng Liu
admin_assigned
2024-10-10T09:26:46.409Z
6569f6cb44ce94a70187f407
/avatars/414ecc9739ed23f9f395bd7cfa65055f.svg
Zicheng Liu
user
ZCLiu35
67073a7a95f90bdb8e86a104
Weiyang Jin
admin_assigned
2024-10-10T09:26:53.527Z
66608add236f958513d21d2e
/avatars/53eca0891c98cbb93be899885160a983.svg
Weiyang Jin
user
Wayne-King
67073a7a95f90bdb8e86a105
Yang Liu
67073a7a95f90bdb8e86a106
Baigui Sun
admin_assigned
2024-10-10T09:27:02.990Z
64d9f1409a6a7ae9846ab8d1
/avatars/208b9d05b30f0094c3301936fabf4656.svg
Baigui Sun
user
sunbaigui
67073a7a95f90bdb8e86a107
Stan Z. Li
admin_assigned
2024-10-10T09:27:08.710Z
64d187bf5de9e1e911d4777f
/avatars/0290d06655340ea57ec3db0b5f89455f.svg
Stan Z. Li
user
szli-0000
Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation
  Learning
This paper delves into the interplay between vision backbones and optimizers,
unvealing an inter-dependent phenomenon termed
\textbf{backbone-optimizer coupling bias}
(BOCB). We observe that canonical CNNs, such as VGG and ResNet, exhibit a
marked co-dependency with SGD families, while recent architectures like ViTs
and ConvNeXt share a tight coupling with the adaptive learning rate ones. We
further show that BOCB can be introduced by both optimizers and certain
backbone designs and may significantly impact the pre-training and downstream
fine-tuning of vision models. Through in-depth empirical analysis, we summarize
takeaways on recommended optimizers and insights into robust vision backbone
architectures. We hope this work can inspire the community to question
long-held assumptions on backbones and optimizers, stimulate further
explorations, and thereby contribute to more robust vision systems. The source
code and models are publicly available at https://bocb-ai.github.io/.
The interplay between vision backbones and optimizers, termed Backbone-Optimizer Coupling Bias, significantly impacts pre-training and fine-tuning of vision models, with different architectures aligning with specific optimizer types.
backbone-optimizer coupling bias VGG ResNet SGD ViTs ConvNeXt adaptive learning rate pre-training downstream fine-tuning