readme: ---
  license: bsd-3-clause
  ---
  ## Model Card: FLAVA
  
  ## Model Details
  
  FLAVA model was developed by the researchers at FAIR to understand if a single model can work across different modalities with a unified architecture. The model was pretrained solely using publicly available multimodal datasets containing 70M image-text pairs in total and thus fully reproducible. Unimodal datasets ImageNet and BookCorpus + CCNews were also used to provide unimodal data to the model. The model (i) similar to CLIP can be used for arbitrary image classification tasks in a zero-shot manner (ii) used for image or text retrieval in a zero-shot manner (iii) can also be fine-tuned for natural language understanding (NLU) tasks such as GLUE and vision-and-language reasoning tasks such as VQA v2. The model is able to use the data available as images, text corpus and image-text pairs. In the original paper, the authors evaluate FLAVA on 32 tasks from computer vision, NLU and vision-and-language domains and show impressive performance across the board scoring higher micro-average than CLIP while being open.
  
  ## Model Date
  Model was originally released in November 2021.
  
  ## Model Type
  
  The FLAVA model uses a ViT-B/32 transformer for both image encoder and text encoder. FLAVA also employs a multimodal encoder on top for multimodal tasks such as vision-and-language tasks (VQA) which is a 6-layer encoder. Each component of FLAVA model can be loaded individually from `facebook/flava-full` checkpoint. If you need complete heads used for pretraining, please use `FlavaForPreTraining` model class otherwise `FlavaModel` should suffice for most use case. This [repository](https://github.com/facebookresearch/multimodal/tree/main/examples/flava) also contains code to pretrain the FLAVA model from scratch.
  
  ## Documents
  
  - [FLAVA Paper](https://arxiv.org/abs/2112.04482)
  
  ## Using with Transformers
  
  ### FlavaModel
  
  FLAVA model supports vision, language and multimodal inputs. You can pass inputs corresponding to the domain you are concerned with to get losses and outputs related to that domain.
  
  ```py
  from PIL import Image
  import requests
  
  from transformers import FlavaProcessor, FlavaModel
  
  model = FlavaModel.from_pretrained("facebook/flava-full")
  processor = FlavaProcessor.from_pretrained("facebook/flava-full")
  
  url = "http://images.cocodataset.org/val2017/000000039769.jpg"
  image = Image.open(requests.get(url, stream=True).raw)
  
  inputs = processor(
    text=["a photo of a cat", "a photo of a dog"], images=[image, image], return_tensors="pt", padding="max_length", max_length=77
  )
  
  outputs = model(**inputs)
  image_embeddings = outputs.image_embeddings # Batch size X (Number of image patches + 1) x Hidden size => 2 X 197 X 768
  text_embeddings = outputs.text_embeddings # Batch size X (Text sequence length + 1) X Hidden size => 2 X 77 X 768
  multimodal_embeddings = outputs.multimodal_embeddings # Batch size X (Number of image patches + Text Sequence Length + 3) X Hidden size => 2 X 275 x 768
  # Multimodal embeddings can be used for multimodal tasks such as VQA
  
  
  ## Pass only image
  from transformers import FlavaFeatureExtractor
  
  feature_extractor = FlavaFeatureExtractor.from_pretrained("facebook/flava-full")
  inputs = feature_extractor(images=[image, image], return_tensors="pt")
  outputs = model(**inputs)
  image_embeddings = outputs.image_embeddings
  
  ## Pass only image
  from transformers import BertTokenizer
  
  tokenizer = BertTokenizer.from_pretrained("facebook/flava-full")
  inputs = tokenizer(["a photo of a cat", "a photo of a dog"], return_tensors="pt", padding="max_length", max_length=77)
  outputs = model(**inputs)
  text_embeddings = outputs.text_embeddings
  ```
  
  #### Encode Image
  
  ```py
  from PIL import Image
  import requests
  
  from transformers import FlavaFeatureExtractor, FlavaModel
  
  model = FlavaModel.from_pretrained("facebook/flava-full")
  feature_extractor = FlavaFeatureExtractor.from_pretrained("facebook/flava-full")
  
  url = "http://images.cocodataset.org/val2017/000000039769.jpg"
  image = Image.open(requests.get(url, stream=True).raw)
  
  inputs = feature_extractor(images=[image], return_tensors="pt")
  
  image_embedding = model.get_image_features(**inputs)
  ```
  
  #### Encode Text
  
  ```py
  from PIL import Image
  
  from transformers import BertTokenizer, FlavaModel
  
  model = FlavaModel.from_pretrained("facebook/flava-full")
  tokenizer = BertTokenizer.from_pretrained("facebook/flava-full")
  
  inputs = tokenizer(text=["a photo of a dog"], return_tensors="pt", padding="max_length", max_length=77)
  
  text_embedding = model.get_text_features(**inputs)
  ```
  
  ### FlavaForPreTraining
  
  FLAVA model supports vision, language and multimodal inputs. You can pass corresponding inputs to modality to get losses and outputs related to that domain.
  
  ```py
  from PIL import Image
  import requests
  
  from transformers import FlavaProcessor, FlavaForPreTraining
  
  model = FlavaForPreTraining.from_pretrained("facebook/flava-full")
  processor = FlavaProcessor.from_pretrained("facebook/flava-full")
  
  url = "http: