readme: ---
  license: apache-2.0
  ---
  
  # Vision-and-Language Transformer (ViLT), pre-trained only
  
  Vision-and-Language Transformer (ViLT) model pre-trained on GCC+SBU+COCO+VG (200k steps). It was introduced in the paper [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334) by Kim et al. and first released in [this repository](https://github.com/dandelin/ViLT). Note: this model only includes the language modeling head.
  
  Disclaimer: The team releasing ViLT did not write a model card for this model so this model card has been written by the Hugging Face team.
  
  ## Intended uses & limitations
  
  You can use the raw model for masked language modeling given an image and a piece of text with [MASK] tokens.
  
  ### How to use
  
  Here is how to use this model in PyTorch:
  
  ```
  from transformers import ViltProcessor, ViltForMaskedLM
  import requests
  from PIL import Image
  import re
  
  url = "http://images.cocodataset.org/val2017/000000039769.jpg"
  image = Image.open(requests.get(url, stream=True).raw)
  text = "a bunch of [MASK] laying on a [MASK]."
  
  processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm")
  model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm")
  
  # prepare inputs
  encoding = processor(image, text, return_tensors="pt")
  
  # forward pass
  outputs = model(**encoding)
  
  tl = len(re.findall("\[MASK\]", text))
  inferred_token = [text]
  
  # gradually fill in the MASK tokens, one by one
  with torch.no_grad():
      for i in range(tl):
          encoded = processor.tokenizer(inferred_token)
          input_ids = torch.tensor(encoded.input_ids).to(device)
          encoded = encoded["input_ids"][0][1:-1]
          outputs = model(input_ids=input_ids, pixel_values=pixel_values)
          mlm_logits = outputs.logits[0]  # shape (seq_len, vocab_size)
          # only take into account text features (minus CLS and SEP token)
          mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :]
          mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1)
          # only take into account text
          mlm_values[torch.tensor(encoded) != 103] = 0
          select = mlm_values.argmax().item()
          encoded[select] = mlm_ids[select].item()
          inferred_token = [processor.decode(encoded)]
  
  selected_token = ""
  encoded = processor.tokenizer(inferred_token)
  processor.decode(encoded.input_ids[0], skip_special_tokens=True)
  ```
  
  ## Training data
  
  (to do)
  
  ## Training procedure
  
  ### Preprocessing
  
  (to do)
  
  ### Pretraining
  
  (to do)
  
  ## Evaluation results
  
  (to do)
  
  ### BibTeX entry and citation info
  
  ```bibtex
  @misc{kim2021vilt,
        title={ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}, 
        author={Wonjae Kim and Bokyung Son and Ildoo Kim},
        year={2021},
        eprint={2102.03334},
        archivePrefix={arXiv},
        primaryClass={stat.ML}
  }
  ```