readme: ---
  #license: not yet
  tags:
  - vision
  datasets:
  - kakaobrain/coyo-labeled-300m
  annotations_creators:
  - machine-generated
  license:
  - apache-2.0
  pretty_name: COYO-Labeled-300M
  task_categories:
  - image-classification
  task_ids:
  - multi-label-image-classification
  - multi-class-image-classification
  inference: false
  ---
  
  # Vision Transformer (large-sized model) 
  
  Vision Transformer (ViT) model pre-trained on [COYO-Labeled-300M](https://github.com/kakaobrain/coyo-dataset/tree/main/subset/COYO-Labeled-300M) (300 million images, 21,841 classes) at resolution 224x224. It was introduced in the paper [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929) by Dosovitskiy et al. However, since the JFT-300M is a private dataset, we tried to reproduce it using the publicly available [COYO-Labeled-300M](https://github.com/kakaobrain/coyo-dataset/tree/main/subset/COYO-Labeled-300M) dataset.
  
  Thanks to Hugging Face team for converting weights of ViT trained in Tensorflow to be used on Pytorch, JAX/Flax and Tensorflow in Hugging Face.
  
  ## Model description
  
  The Vision Transformer (ViT) is a transformer model pretrained on a large collection of images in a supervised fashion, namely [COYO-Labeled-300M](https://github.com/kakaobrain/coyo-dataset/tree/main/subset/COYO-Labeled-300M), at a resolution of 224x224 pixels. 
  
  Images are presented to the model as a sequence of fixed-size patches (resolution 16x16), which are linearly embedded. One also adds a [CLS] token to the beginning of a sequence to use it for classification tasks. One also adds absolute position embeddings before feeding the sequence to the layers of the Transformer.
  
  This ViT model is pretrained with COYO-Labeled-300M at resolution 224x224. Please see details [here](https://github.com/kakaobrain/coyo-vit)
  
  ## Intended uses & limitations
  
  You can use weights from ViT models for image classification, downstream. Codes for reproduction are also provided. Please see this [github repository](https://github.com/kakaobrain/coyo-vit) for pretraining and finetuning code.
  
  ### How to use
  
  Here is how to use this model in PyTorch:
  
  ```python
  WIP
  ```
  
  Here is how to use this model in JAX/Flax:
  
  ```python
  WIP
  ```
  
  Here is how to use this model in Tensorflow:
  
  ```python
  WIP
  ```
  
  ## Training data
  
  The ViT model was pretrained on [COYO-Labeled-300M](https://github.com/kakaobrain/coyo-dataset/tree/main/subset/COYO-Labeled-300M), a dataset consisting of 300 million images and 21k classes. 
  
  ## Training procedure
  
  ### Preprocessing
  
  The exact details of preprocessing of images during training/validation can be found [here](https://github.com/kakaobrain/coyo-vit). 
  
  Images are inception-cropped to the same resolution (224x224) and normalized across the RGB channels with mean (0.5, 0.5, 0.5) and standard deviation (0.5, 0.5, 0.5).
  
  ### Pretraining
  
  The model was trained on TPUv3 hardware. All model variants are trained with a batch size of 4096 and learning rate warmup of 10k steps. Pre-training resolution is 224.
  More detail, please see [here](https://github.com/kakaobrain/coyo-vit)
  
  ### Evaluation results
  
  | Model        | Upstream Dataset      | Resolution     | ImageNet (downstream)     | ImageNet-ReaL (dwonstream)     | Public     |
  |----------    |-------------------    |------------    |-----------------------    |----------------------------    |--------    |
  | ViT-L/16     | JFT-300M              | 512            | 87.76                     | 90.54                          | X          |
  | ViT-L/16     | COYO-Labeled-300M     | 512            | 87.24 (-0.52)             | 90.03 (-0.51)                  | O          |
  | ViT-L/16     | JFT-300M              | 384            | 87.12                     | 89.99                          | X          |
  | ViT-L/16     | COYO-Labeled-300M     | 384            | 86.72 (-0.4)              | 89.84 (-0.15)                  | O          |
  
  ## Citation
  ```bibtex
  @misc{kakaobrain2022coyo-vit,
    title         = {COYO-ViT},
    author        = {Lee, Sungjun and Park, Beomhee},
    year          = {2022},
    howpublished  = {\url{https://github.com/kakaobrain/coyo-vit}},
  }
  ```
  ```bibtex
  @misc{kakaobrain2022coyo-700m,
    title         = {COYO-700M: Image-Text Pair Dataset},
    author        = {Byeon, Minwoo and Park, Beomhee and Kim, Haecheon and Lee, Sungjun and Baek, Woonhyuk and Kim, Saehoon},
    year          = {2022},
    howpublished  = {\url{https://github.com/kakaobrain/coyo-dataset}},
  }
  ```
  ```bibtex
  @misc{dosovitskiy2020image,
      title   = {An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
      author  = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
      year    = {2020},
      eprint  = {2010.11929},
      archivePrefix = {arXiv},
      primaryClass = {cs.CV}
  }
  ```
  
  ## License
  The 