POA: Pre-training Once for Models of All Sizes
Large-scale self-supervised pre-training has paved the way for one foundation
model to handle many different vision tasks. Most pre-training methodologies
train a single model of a certain size at one time. Nevertheless, various
computation or storage constraints in real-world scenarios require substantial
efforts to develop a series of models with different sizes to deploy. Thus, in
this study, we propose a novel tri-branch self-supervised training framework,
termed as POA (Pre-training Once for All), to tackle this aforementioned issue.
Our approach introduces an innovative elastic student branch into a modern
self-distillation paradigm. At each pre-training step, we randomly sample a
sub-network from the original student to form the elastic student and train all
branches in a self-distilling fashion. Once pre-trained, POA allows the
extraction of pre-trained models of diverse sizes for downstream tasks.
Remarkably, the elastic student facilitates the simultaneous pre-training of
multiple models with different sizes, which also acts as an additional ensemble
of models of various sizes to enhance representation learning. Extensive
experiments, including k-nearest neighbors, linear probing evaluation and
assessments on multiple downstream tasks demonstrate the effectiveness and
advantages of our POA. It achieves state-of-the-art performance using ViT, Swin
Transformer and ResNet backbones, producing around a hundred models with
different sizes through a single pre-training session. The code is available
at: https://github.com/Qichuzyy/POA.
66b039c1f0bb529189680c93
Yingying Zhang
claimed_verified
2024-08-05T09:32:28.705Z
66b098fd5a6f5c9d93ed87e7
/avatars/5822995330d1ab47f1a92c1ffc04ac82.svg
Yingying Zhang
user
qichu7331
66b039c1f0bb529189680c94
Xin Guo
claimed_verified
2024-08-06T07:44:05.598Z
619b17fb688ab0c91019e449
https://cdn-avatars.huggingface.co/v1/production/uploads/619b17fb688ab0c91019e449/hWdAhHZPSSqCg4GChmjCd.jpeg
jamesguo
user
jamesguo
66b039c1f0bb529189680c95
Jiangwei Lao
admin_assigned
2024-08-05T09:03:19.376Z
65893a7415b65eb9ba006d37
/avatars/cc997a1615deefb84469aa702edb89ca.svg
Lao Jiangwei
user
LaoJiangwei
66b039c1f0bb529189680c96
Lei Yu
66b039c1f0bb529189680c97
Lixiang Ru
admin_assigned
2024-08-05T09:04:17.951Z
63da06b8aefaa73525883b6b
/avatars/29fa27c3fd2ec5f96583400b64282540.svg
Lixiang Ru
user
rulixiang
66b039c1f0bb529189680c98
Jian Wang
66b039c1f0bb529189680c99
Guo Ye
66b039c1f0bb529189680c9a
Huimei He
66b039c1f0bb529189680c9b
Jingdong Chen
admin_assigned
2024-08-05T09:05:19.952Z
64575ac8cd935d48a47774ec
/avatars/5d211e2c13d6c4e011e5e58b738413f7.svg
chenjingdong 
user
chenjingdong
66b039c1f0bb529189680c9c
Ming Yang
POA: Pre-training Once for Models of All Sizes
Large-scale self-supervised pre-training has paved the way for one foundation
model to handle many different vision tasks. Most pre-training methodologies
train a single model of a certain size at one time. Nevertheless, various
computation or storage constraints in real-world scenarios require substantial
efforts to develop a series of models with different sizes to deploy. Thus, in
this study, we propose a novel tri-branch self-supervised training framework,
termed as POA (Pre-training Once for All), to tackle this aforementioned issue.
Our approach introduces an innovative elastic student branch into a modern
self-distillation paradigm. At each pre-training step, we randomly sample a
sub-network from the original student to form the elastic student and train all
branches in a self-distilling fashion. Once pre-trained, POA allows the
extraction of pre-trained models of diverse sizes for downstream tasks.
Remarkably, the elastic student facilitates the simultaneous pre-training of
multiple models with different sizes, which also acts as an additional ensemble
of models of various sizes to enhance representation learning. Extensive
experiments, including k-nearest neighbors, linear probing evaluation and
assessments on multiple downstream tasks demonstrate the effectiveness and
advantages of our POA. It achieves state-of-the-art performance using ViT, Swin
Transformer and ResNet backbones, producing around a hundred models with
different sizes through a single pre-training session. The code is available
at: https://github.com/Qichuzyy/POA.
POA framework facilitates the pre-training of various-sized models through an innovative elastic student branch within self-distillation, achieving state-of-the-art performance across multiple backbones.
self-supervised pre-training tri-branch framework POA elastic student branch self-distillation k-nearest neighbors linear probing evaluation downstream tasks ViT Swin Transformer ResNet ensemble learning representation learning