title: POA: Pre-training Once for Models of All Sizes
summary: Large-scale self-supervised pre-training has paved the way for one foundation
  model to handle many different vision tasks. Most pre-training methodologies
  train a single model of a certain size at one time. Nevertheless, various
  computation or storage constraints in real-world scenarios require substantial
  efforts to develop a series of models with different sizes to deploy. Thus, in
  this study, we propose a novel tri-branch self-supervised training framework,
  termed as POA (Pre-training Once for All), to tackle this aforementioned issue.
  Our approach introduces an innovative elastic student branch into a modern
  self-distillation paradigm. At each pre-training step, we randomly sample a
  sub-network from the original student to form the elastic student and train all
  branches in a self-distilling fashion. Once pre-trained, POA allows the
  extraction of pre-trained models of diverse sizes for downstream tasks.
  Remarkably, the elastic student facilitates the simultaneous pre-training of
  multiple models with different sizes, which also acts as an additional ensemble
  of models of various sizes to enhance representation learning. Extensive
  experiments, including k-nearest neighbors, linear probing evaluation and
  assessments on multiple downstream tasks demonstrate the effectiveness and
  advantages of our POA. It achieves state-of-the-art performance using ViT, Swin
  Transformer and ResNet backbones, producing around a hundred models with
  different sizes through a single pre-training session. The code is available
  at: https://github.com/Qichuzyy/POA.
paper_authors._id: 66b039c1f0bb529189680c93
paper_authors.name: Yingying Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-05T09:32:28.705Z
paper_authors._id: 66b098fd5a6f5c9d93ed87e7
paper_authors.avatarUrl: /avatars/5822995330d1ab47f1a92c1ffc04ac82.svg
paper_authors.fullname: Yingying Zhang
paper_authors.type: user
paper_authors.user: qichu7331
paper_authors._id: 66b039c1f0bb529189680c94
paper_authors.name: Xin Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-06T07:44:05.598Z
paper_authors._id: 619b17fb688ab0c91019e449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619b17fb688ab0c91019e449/hWdAhHZPSSqCg4GChmjCd.jpeg
paper_authors.fullname: jamesguo
paper_authors.type: user
paper_authors.user: jamesguo
paper_authors._id: 66b039c1f0bb529189680c95
paper_authors.name: Jiangwei Lao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:03:19.376Z
paper_authors._id: 65893a7415b65eb9ba006d37
paper_authors.avatarUrl: /avatars/cc997a1615deefb84469aa702edb89ca.svg
paper_authors.fullname: Lao Jiangwei
paper_authors.type: user
paper_authors.user: LaoJiangwei
paper_authors._id: 66b039c1f0bb529189680c96
paper_authors.name: Lei Yu
paper_authors._id: 66b039c1f0bb529189680c97
paper_authors.name: Lixiang Ru
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:04:17.951Z
paper_authors._id: 63da06b8aefaa73525883b6b
paper_authors.avatarUrl: /avatars/29fa27c3fd2ec5f96583400b64282540.svg
paper_authors.fullname: Lixiang Ru
paper_authors.type: user
paper_authors.user: rulixiang
paper_authors._id: 66b039c1f0bb529189680c98
paper_authors.name: Jian Wang
paper_authors._id: 66b039c1f0bb529189680c99
paper_authors.name: Guo Ye
paper_authors._id: 66b039c1f0bb529189680c9a
paper_authors.name: Huimei He
paper_authors._id: 66b039c1f0bb529189680c9b
paper_authors.name: Jingdong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:05:19.952Z
paper_authors._id: 64575ac8cd935d48a47774ec
paper_authors.avatarUrl: /avatars/5d211e2c13d6c4e011e5e58b738413f7.svg
paper_authors.fullname: chenjingdong 
paper_authors.type: user
paper_authors.user: chenjingdong
paper_authors._id: 66b039c1f0bb529189680c9c
paper_authors.name: Ming Yang
paper_title: POA: Pre-training Once for Models of All Sizes
paper_summary: Large-scale self-supervised pre-training has paved the way for one foundation
  model to handle many different vision tasks. Most pre-training methodologies
  train a single model of a certain size at one time. Nevertheless, various
  computation or storage constraints in real-world scenarios require substantial
  efforts to develop a series of models with different sizes to deploy. Thus, in
  this study, we propose a novel tri-branch self-supervised training framework,
  termed as POA (Pre-training Once for All), to tackle this aforementioned issue.
  Our approach introduces an innovative elastic student branch into a modern
  self-distillation paradigm. At each pre-training step, we randomly sample a
  sub-network from the original student to form the elastic student and train all
  branches in a self-distilling fashion. Once pre-trained, POA allows the
  extraction of pre-trained models of diverse sizes for downstream tasks.
  Remarkably, the elastic student facilitates the simultaneous pre-training of
  multiple models with different sizes, which also acts as an additional ensemble
  of models of various sizes to enhance representation learning. Extensive
  experiments, including k-nearest neighbors, linear probing evaluation and
  assessments on multiple downstream tasks demonstrate the effectiveness and
  advantages of our POA. It achieves state-of-the-art performance using ViT, Swin
  Transformer and ResNet backbones, producing around a hundred models with
  different sizes through a single pre-training session. The code is available
  at: https://github.com/Qichuzyy/POA.
paper_ai_summary: POA framework facilitates the pre-training of various-sized models through an innovative elastic student branch within self-distillation, achieving state-of-the-art performance across multiple backbones.
paper_ai_keywords: self-supervised pre-training tri-branch framework POA elastic student branch self-distillation k-nearest neighbors linear probing evaluation downstream tasks ViT Swin Transformer ResNet ensemble learning representation learning