readme: ---
  license: mit
  ---
  # Model Card for CLIP-convnext_large_d.laion2B-s26B-b102K-augreg
  
  #  Table of Contents
  
  1. [Model Details](#model-details)
  2. [Uses](#uses)
  3. [Training Details](#training-details)
  4. [Evaluation](#evaluation)
  5. [Acknowledgements](#acknowledgements)
  6. [Citation](#citation)
  
  # Model Details
  
  ## Model Description
  
  A series of CLIP [ConvNeXt-Large](https://arxiv.org/abs/2201.03545) (w/ extra text depth, vision MLP head) models trained on subsets [LAION-5B](https://arxiv.org/abs/2210.08402) using [OpenCLIP](https://github.com/mlfoundations/open_clip).
  
  Goals:
    * Explore an alternative to ViT and ResNet (w/ AttentionPooling) CLIP models that scales well with model size and image resolution
  
  Firsts:
    * First known ConvNeXt CLIP models trained at scale in the range of CLIP ViT-L/16, ViT-L14, and RN50x16
    * First released model weights exploring increase of augmentation + regularization for image tower via adding (greater scale range of RRC, random erasing, stochastic depth)
  
  The models utilize:
    * the [timm](https://github.com/rwightman/pytorch-image-models) ConvNeXt-Large model (`convnext_large`) as the image tower
    * a MLP (`fc - gelu - drop - fc`) head in vision tower instead of the single projection of other CLIP models
    * a text tower with same width but 4 layers more depth than ViT-L / RN50x16 models (depth 16, embed dim 768).
    
  The models are trained at 256x256 (working on 384 variants) image resolution.
  
  At 256x256, the ConvNext-Large-D used roughly 1/2 the training FLOPs to achieve accuracy greater than previous L/14 model trained on LAION-2B. L/14 model is ~1.65x more GMAC, 1.45x more activations, and 1.22x more parameters. The ConvNeXt was trained with 26B samples-seen and L/14 with 34B.
  
  
  | Model | Dataset | Resolution | AugReg | Top-1 ImageNet Zero-Shot (%) |
  | ----- | ------- | ---------- | ------------ | --------- |
  | [convnext_large_d.laion2b_s26b_b102k-augreg](https://huggingface.co/laion/CLIP-convnext_large_d.laion2B-s26B-b102K-augreg) | LAION-2B | 256x256 |  RRC (0.33, 1.0), RE (0.35), SD (0.1), D(0.1) | 75.9 |
  
  RRC = Random Resize Crop (crop pcts), RE = Random Erasing (prob), SD = Stochastic Depth (prob) -- image tower only, D = Dropout (prob) -- image tower head only
  
  LAION-A = LAION Aesthetic, an ~900M sample subset of LAION-2B with pHash dedupe and asthetic score filtering.
  
  Model training done by Ross Wightman on the [stability.ai](https://stability.ai/) cluster.
  
  # Uses
  
  As per the original [OpenAI CLIP model card](https://github.com/openai/CLIP/blob/d50d76daa670286dd6cacf3bcd80b5e4823fc8e1/model-card.md), this model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such model. 
  
  The OpenAI CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis. Additionally, the LAION-5B blog (https://laion.ai/blog/laion-5b/) and upcoming paper include additional discussion as it relates specifically to the training dataset. 
  
  ## Direct Use
  
  Zero-shot image classification, image and text retrieval, among others.
  
  ## Downstream Use
  
  Image classification and other image task fine-tuning, linear probe image classification, image generation guiding and conditioning, among others.
  
  ## Out-of-Scope Use
  
  As per the OpenAI models,
  
  **Any** deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful. 
  
  Certain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.
  
  Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.
  
  Further the above notice, the LAION-5B dataset used in training of these models has additional considerations, see below.
  
  # Training Details
  
  ## Training Data
  
  This model was trained with one of (see table in intro):
  * LAION-2B - A 2 billion sample English subset of LAION-5B (https://laion.ai/blog/laion-5b/).
  * LAION-Aesthetic - A 900M sample subset of LAION-2B with pHash dedupe and asthetic score filtering
  
  **IMPORTANT NOTE:** The motivatio