Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities
<|reference_start|>Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities: Recent advancements in Vision-Language (VL) models have sparked interest in their deployment on edge devices, yet challenges in handling diverse visual modalities, manual annotation, and computational constraints remain. We introduce EdgeVL, a novel framework that bridges this gap by seamlessly integrating dual-modality knowledge distillation and quantization-aware contrastive learning. This approach enables the adaptation of large VL models, like CLIP, for efficient use with both RGB and non-RGB images on resource-limited devices without the need for manual annotations. EdgeVL not only transfers visual language alignment capabilities to compact models but also maintains feature quality post-quantization, significantly enhancing open-vocabulary classification performance across various visual modalities. Our work represents the first systematic effort to adapt large VL models for edge deployment, showcasing up to 15.4% accuracy improvements on multiple datasets and up to 93-fold reduction in model size.<|reference_end|>
@article{cai2024self-adapting,
    title={Self-Adapting Large Visual-Language Models to Edge Devices across Visual
  Modalities},
    author={Kaiwen Cai, Zhekai Duan, Gaowen Liu, Charles Fleming, and Chris
  Xiaoxuan Lu},
    journal={arXiv preprint arXiv:2403.04908},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04908},
    primaryClass={cs.CV}
}