By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting
<|reference_start|>By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting: Large language models (LLMs) have demonstrated exceptional abilities across various domains. However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences. We propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs). We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions. Additionally, we introduce a visualization generator that automates the creation of optimal visualizations tailored to a given sensory task, eliminating the need for prior task-specific knowledge. We evaluated our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy than text-based prompts and reducing token costs by 15.8 times. Our findings highlight the effectiveness and cost-efficiency of visual prompts with MLLMs for various sensory tasks. The source code is available at https://github.com/diamond264/ByMyEyes.<|reference_end|>
@article{yoon2024by,
    title={By My Eyes: Grounding Multimodal Large Language Models with Sensor Data
  via Visual Prompting},
    author={Hyungjun Yoon, Biniyam Aschalew Tolera, Taesik Gong, Kimin Lee,
  Sung-Ju Lee},
    journal={arXiv preprint arXiv:2407.10385},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.10385},
    primaryClass={cs.CL cs.AI cs.CV cs.LG}
}