readme: # Deep Model Assembling
  
  This repository contains the pre-trained models for [Deep Model Assembling](https://arxiv.org/abs/2212.04129).
  
  > **Title**:&emsp;&emsp;[**Deep Model Assembling**](https://arxiv.org/abs/2212.04129)  
  > **Authors**:&nbsp;&nbsp;[Zanlin Ni](https://scholar.google.com/citations?user=Yibz_asAAAAJ&hl=en&oi=ao), [Yulin Wang](https://scholar.google.com/citations?hl=en&user=gBP38gcAAAAJ), Jiangwei Yu, [Haojun Jiang](https://scholar.google.com/citations?hl=en&user=ULmStp8AAAAJ), [Yue Cao](https://scholar.google.com/citations?hl=en&user=iRUO1ckAAAAJ), [Gao Huang](https://scholar.google.com/citations?user=-P9LwcgAAAAJ&hl=en&oi=ao) (Corresponding Author)  
  > **Institute**: Tsinghua University and Beijing Academy of Artificial Intelligence (BAAI)  
  > **Publish**:&nbsp;&nbsp;&nbsp;*arXiv preprint ([arXiv 2212.04129](https://arxiv.org/abs/2212.04129))*  
  > **Contact**:&nbsp;&nbsp;nzl22 at mails dot tsinghua dot edu dot cn
  
  ## Overview
  
  In this paper, we present a divide-and-conquer strategy for training large models. Our algorithm, Model Assembling, divides a large model into smaller modules, optimizes them independently, and then assembles them together. Though conceptually simple, our method significantly outperforms end-to-end (E2E) training in terms of both training efficiency and final accuracy. For example, on ViT-H, Model Assembling outperforms E2E training by **2.7%**, while reducing the training cost by **43%**.
  
  ## Models
  
  | model | image size | #param. | top-1 acc. | checkpoint                                                                                                  |
  | ----- | ---------- | ------- | ---------- | ----------------------------------------------------------------------------------------------------------- |
  | ViT-B | 224x224    | 87M     | 82.4%      | [🤗 HF   link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/vit_base.pth)           |
  | ViT-B | 384x384    | 87M     | 84.2%      | [🤗 HF   link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/high_res/vit_base.pth)  |
  | ViT-L | 224x224    | 304M    | 83.9%      | [🤗 HF   link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/vit_large.pth)          |
  | ViT-L | 384x384    | 304M    | 85.3%      | [🤗 HF   link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/high_res/vit_large.pth) |
  | ViT-H | 224x224    | 632M    | 84.3%      | [🤗 HF   link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/vit_huge.pth)           |
  | ViT-H | 392x392    | 632M    | 85.6%      | [🤗 HF   link](https://huggingface.co/nzl-thu/Model-Assembling/blob/main/pretrained/high_res/vit_huge.pth)  |
  
  ## Data Preparation
  
  - The ImageNet dataset should be prepared as follows:
  
  ```
  data
  ├── train
  │   ├── folder 1 (class 1)
  │   ├── folder 2 (class 1)
  │   ├── ...
  ├── val
  │   ├── folder 1 (class 1)
  │   ├── folder 2 (class 1)
  │   ├── ...
  
  ```
  
  ## Citation
  
  If you find our work helpful, please **star🌟** this repo and **cite📑** our paper. Thanks for your support!
  
  ```
  @article{Ni2022Assemb,
    title={Deep Model Assembling},
    author={Ni, Zanlin and Wang, Yulin and Yu, Jiangwei and Jiang, Haojun and Cao, Yue and Huang, Gao},
    journal={arXiv preprint arXiv:2212.04129},
    year={2022}
  }
  ```
  
  ## Acknowledgements
  
  Our implementation is mainly based on [deit](https://github.com/facebookresearch/deit). We thank to their clean codebase.
  
  ## Contact
  
  If you have any questions or concerns, please send mail to [nzl22@mails.tsinghua.edu.cn](mailto:nzl22@mails.tsinghua.edu.cn).
  