transformers safetensors vlm text-generation image-captioning visual-question-answering image-to-text en dataset:sbu_captions dataset:visual_genome dataset:HuggingFaceM4/VQAv2 dataset:ChristophSchuhmann/MS_COCO_2017_URL_TEXT base_model:unum-cloud/uform-vl-english base_model:finetune:unum-cloud/uform-vl-english license:apache-2.0 autotrain_compatible endpoints_compatible region:us
---
base_model: unum-cloud/uform-vl-english
datasets:
- sbu_captions
- visual_genome
- HuggingFaceM4/VQAv2
- ChristophSchuhmann/MS_COCO_2017_URL_TEXT
language:
- en
license: apache-2.0
pipeline_tag: image-to-text
tags:
- image-captioning
- visual-question-answering
widget:
- src: preview-interior.png
  output:
    text: The living room is cozy, featuring a red leather chair and a white table.
      The chair is in the center, and the table is on the left side. A lamp on the
      left side illuminates the space. A large picture hangs on the wall, adding artistic
      flair. A vase on the table adds a decorative touch. The room is well-lit, creating
      a warm and inviting atmosphere.
- src: preview-girl.png
  output:
    text: A young girl stands in a grassy field, holding an umbrella to shield herself
      from the rain. She dons a yellow dress and seems to relish her time outdoors.
      The umbrella is open, offering protection from the rain. The field is bordered
      by trees, fostering a tranquil and natural ambiance
---
<Gallery />

<h1 align="center">UForm</h1>
<h3 align="center">
Pocket-Sized Multimodal AI<br/>
For Content Understanding and Generation<br/>
</h3>

## Description 

UForm-Gen is a small generative vision-language model primarily designed for Image Captioning and Visual Question Answering. The model consists of two parts: 

1. [`uform-vl-english`](https://huggingface.co/unum-cloud/uform-vl-english) visual encoder,
2. [`Sheared-LLaMA-1.3B`](https://huggingface.co/princeton-nlp/Sheared-LLaMA-1.3B) language model tuned on instruction datasets.

The model was pre-trained on: MSCOCO, SBU Captions, Visual Genome, VQAv2, GQA and a few internal datasets.

### Usage

```bash
pip install uform
```

The generative model can be used to caption images, summarize their content, or answer questions about them.
The exact behavior is controlled by prompts.

```python
from uform.gen_model import VLMForCausalLM, VLMProcessor

model = VLMForCausalLM.from_pretrained("unum-cloud/uform-gen")
processor = VLMProcessor.from_pretrained("unum-cloud/uform-gen")

# [cap] Narrate the contents of the image with precision.
# [cap] Summarize the visual content of the image.
# [vqa] What is the main subject of the image?
prompt = "[cap] Summarize the visual content of the image."
image = Image.open("zebra.jpg")

inputs = processor(texts=[prompt], images=[image], return_tensors="pt")
with torch.inference_mode():
     output = model.generate(
        **inputs,
        do_sample=False,
        use_cache=True,
        max_new_tokens=128,
        eos_token_id=32001,
        pad_token_id=processor.tokenizer.pad_token_id
    )

prompt_len = inputs["input_ids"].shape[1]
decoded_text = processor.batch_decode(output[:, prompt_len:])[0]
```


## Evaluation

For captioning evaluation we measure CLIPScore and RefCLIPScore¹.

| Model                               | Size | Caption Length | CLIPScore | RefCLIPScore |
| :---------------------------------- | ---: | -------------: | --------: | -----------: |
| `llava-hf/llava-1.5-7b-hf`          |   7B |           Long |     0.878 |        0.529 |
| `llava-hf/llava-1.5-7b-hf`          |   7B |          Short |     0.886 |        0.531 |
|                                     |
| `Salesforce/instructblip-vicuna-7b` |   7B |           Long |     0.902 |        0.534 |
| `Salesforce/instructblip-vicuna-7b` |   7B |          Short |     0.848 |        0.523 |
|                                     |                                                  |
| `unum-cloud/uform-gen`              | 1.5B |           Long |     0.847 |        0.523 |
| `unum-cloud/uform-gen`              | 1.5B |          Short |     0.842 |        0.522 |

Results for VQAv2 evaluation.

| Model                      | Size | Accuracy |
| :------------------------- | ---: | -------: |
| `llava-hf/llava-1.5-7b-hf` |   7B |     78.5 |
| `unum-cloud/uform-gen`     | 1.5B |     66.5 |

¹ We used `apple/DFN5B-CLIP-ViT-H-14-378` CLIP model.


## Speed

On RTX 3090, the following performance is expected on text token generation using `float16`, equivalent PyTorch settings, and greedy decoding.

| Model                               | Size |               Speed |   Speedup |
| :---------------------------------- | ---: | ------------------: | --------: |
| `llava-hf/llava-1.5-7b-hf`          |   7B |  ~ 40 tokens/second |           |
| `Salesforce/instructblip-vicuna-7b` |   7B |  ~ 40 tokens/second |           |
| `unum-cloud/uform-gen`              | 1.5B | ~ 140 tokens/second | __x 3.5__ |

