litert-community/SmolVLM-256M-Instruct

This model provides HuggingFaceTB/SmolVLM-256M-Instruct model in TFLite format.
You can use this model with AI Edge Cpp Example. You need to slightly modify cpp pipeline, just send image tensor as input (see COLAB example below).
Please note that, at the moment, AI Edge Torch VLMS not supported on MediaPipe LLM Inference API, for example qwen_vl model, that was used as reference to write SmolVLM-256M-Instruct convertation scripts.

Use the models

Colab

TFlite convertation

To fine-tune SmolVLM on a specific task, you can follow the fine-tuning tutorial.
Than, you can convert model to TFlite using custom smalvlm scripts (see Readme.md).
You can also check the official documentation ai-edge-torch generative.

Details

The model was converted with the following parameters:

python convert_to_tflite.py --quantize="dynamic_int8"\
 --checkpoint_path='./models/SmolVLM-256M-Instruct' --output_path="./models/SmolVLM-256M-Instruct-tflite"\
 --mask_as_input=True --prefill_seq_lens=256 --kv_cache_max_len=2048