readme: ---
  license: apache-2.0
  datasets:
  - hl-scenes
  
  library_name: pytorch
  tags:
  - pytorch
  - image-to-text
  ---
  
  # Model Card: VinVL for Captioning 🖼️
  
  [Microsoft's VinVL](https://github.com/microsoft/Oscar) base fine-tuned on [HL-scenes]() dataset for **scene description generation** downstream task.
  
  # Model fine-tuning 🏋️‍
  
  The model has been finetuned for 10 epochs on the scenes captions of the [HL]() dataset (available on 🤗 HUB: [michelecafagna26/hl](https://huggingface.co/datasets/michelecafagna26/hl))
  
  # Test set metrics 📈
  
  Obtained with beam size 5 and max length 20
  
  | Bleu-1 | Bleu-2 | Bleu-3 | Bleu-4 | METEOR | ROUGE-L | CIDEr | SPICE |
  |--------|--------|--------|--------|--------|---------|-------|-------|
  |  0.68  |  0.55  |  0.45  |  0.36  |  0.36  |  0.63   |  1.42 |  0.40 |
  
  
  # Usage and Installation:
  
  More info about how to install and use this model can be found here: [michelecafagna26/VinVL
  ](https://github.com/michelecafagna26/VinVL)
  
  # Feature extraction ⛏️
  
  This model has a separate Visualbackbone used to extract features.
  More info about:
  - the model here: [michelecafagna26/vinvl_vg_x152c4](https://huggingface.co/michelecafagna26/vinvl_vg_x152c4)
  - the usage here [michelecafagna26/vinvl-visualbackbone](https://github.com/michelecafagna26/vinvl-visualbackbone)
  
  # Quick start: 🚀
  
  ```python
  from transformers.pytorch_transformers import BertConfig, BertTokenizer
  from oscar.modeling.modeling_bert import BertForImageCaptioning
  from oscar.wrappers import OscarTensorizer
  
  ckpt = "path/to/the/checkpoint"
  device = "cuda" if torch.cuda.is_available() else "cpu"
  
  # original code
  config = BertConfig.from_pretrained(ckpt)
  tokenizer = BertTokenizer.from_pretrained(ckpt)
  model = BertForImageCaptioning.from_pretrained(ckpt, config=config).to(device)
  
  # This takes care of the preprocessing
  tensorizer = OscarTensorizer(tokenizer=tokenizer, device=device)
  
  # numpy-arrays with shape (1, num_boxes, feat_size)
  # feat_size is 2054 by default in VinVL
  visual_features = torch.from_numpy(feat_obj).to(device).unsqueeze(0)
  
  # labels are usually extracted by the features extractor
  labels = [['boat', 'boat', 'boat', 'bottom', 'bush', 'coat', 'deck', 'deck', 'deck', 'dock', 'hair', 'jacket']]
  
  inputs = tensorizer.encode(visual_features, labels=labels)
  outputs = model(**inputs)
  
  pred = tensorizer.decode(outputs)
  
  # the output looks like this:
  # pred = {0: [{'caption': 'in a library', 'conf': 0.7070220112800598]}
  ```
  
  # Citations 🧾
  
  This is the model used in:
  
  ```BibTeX
  @misc{cafagna2022understanding,
    author = {Cafagna Michele and van Deemter Kees and Gatt Albert},
    title = {Understanding Cross-modal Interactions in V&amp;L Models that Generate Scene Descriptions},
    doi = {10.48550/ARXIV.2211.04971},
    url = {https://arxiv.org/abs/2211.04971},
    keywords = {Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
  }
  
  ```
  
  
  Please consider citing the original project and the VinVL paper
  
  ```BibTeX
  
  @misc{han2021image,
        title={Image Scene Graph Generation (SGG) Benchmark}, 
        author={Xiaotian Han and Jianwei Yang and Houdong Hu and Lei Zhang and Jianfeng Gao and Pengchuan Zhang},
        year={2021},
        eprint={2107.12604},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
  }
  
  @inproceedings{zhang2021vinvl,
    title={Vinvl: Revisiting visual representations in vision-language models},
    author={Zhang, Pengchuan and Li, Xiujun and Hu, Xiaowei and Yang, Jianwei and Zhang, Lei and Wang, Lijuan and Choi, Yejin and Gao, Jianfeng},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    pages={5579--5588},
    year={2021}
  }
  
  ```
  