SmolVLM2-500M-Video-Instruct-Int8

This version of SmolVLM2-500M-Video-Instruct has been converted to run on the Axera NPU using w8a16 quantization.

Compatible with Pulsar2 version: 4.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo:

• https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct

• Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

Support Platform

• AX650
  • M4N-Dock(爱芯派Pro)

How to use

Download all files from this repository to the device.

Using AX650 Board

ai@ai-bj ~/yongqiang/SmolVLM2-500M-Video-Instruct $ tree -L 1
.
├── assets
├── embeds
├── infer_axmodel.py
├── README.md
├── smolvlm2_axmodel
├── smolvlm2_tokenizer
└── vit_mdoel

5 directories, 2 files

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

Multimodal Understanding

input image

input text:

Can you describe this image?

log information:

ai@ai-bj ~/yongqiang/SmolVLM2-500M-Video-Instruct $ python3 infer_axmodel.py

input prompt: Can you describe this image?

answer >>  The image captures a close-up view of a pink flower, prominently featuring a bumblebee. The bumblebee, with its black and yellow stripes, is in the center of the frame, its body slightly tilted to the left. The flower, with its petals fully spread, is the main subject of the image. The background is blurred, drawing focus to the flower and the bumblebee. The blurred background suggests a garden or a field, providing a sense of depth to the image. The^@ colors in the image are vibrant, with the pink of the flower contrasting against the green of the leaves and the brown of the stems. The image does not provide enough detail to confidently identify the specific location or landmark referred to as "sa_16743".