---
license: bsd-3-clause
language:
  - en
  - zh
base_model:
  - HuggingFaceTB/SmolVLM2-500M-Video-Instruct
pipeline_tag: visual-question-answering
tags:
  - HuggingFaceTB
  - SmolVLM2-500M-Video-Instruct
---

# SmolVLM2-500M-Video-Instruct-Int8

This version of SmolVLM2-500M-Video-Instruct has been converted to run on the Axera NPU using **w8a16** quantization.

Compatible with Pulsar2 version: 4.0

## Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo:
- https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct

<!-- - [Github for SmolVLM2-500M-Video-Instruct.axera](https://github.com/AXERA-TECH/SmolVLM2-500M-Video-Instruct.axera) -->
- [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html)

## Support Platform
- AX650
  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)

<!-- ## TODO Model infer time -->

## How to use

Download all files from this repository to the device.

**Using AX650 Board**

```bash
ai@ai-bj ~/yongqiang/SmolVLM2-500M-Video-Instruct $ tree -L 1
.
├── assets
├── embeds
├── infer_axmodel.py
├── README.md
├── smolvlm2_axmodel
├── smolvlm2_tokenizer
└── vit_mdoel

5 directories, 2 files
```

#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

**Multimodal Understanding**

input image

![](assets/bee.jpg)

input text:

```
Can you describe this image?
```

log information:

```bash
ai@ai-bj ~/yongqiang/SmolVLM2-500M-Video-Instruct $ python3 infer_axmodel.py

input prompt: Can you describe this image?

answer >>  The image captures a close-up view of a pink flower, prominently featuring a bumblebee. The bumblebee, with its black and yellow stripes, is in the center of the frame, its body slightly tilted to the left. The flower, with its petals fully spread, is the main subject of the image. The background is blurred, drawing focus to the flower and the bumblebee. The blurred background suggests a garden or a field, providing a sense of depth to the image. The^@ colors in the image are vibrant, with the pink of the flower contrasting against the green of the leaves and the brown of the stems. The image does not provide enough detail to confidently identify the specific location or landmark referred to as "sa_16743".
```