MultiSurf-GPT: Facilitating Context-Aware Reasoning with Large-Scale Language Models for Multimodal Surface Sensing
<|reference_start|>MultiSurf-GPT: Facilitating Context-Aware Reasoning with Large-Scale Language Models for Multimodal Surface Sensing: Surface sensing is widely employed in health diagnostics, manufacturing and safety monitoring. Advances in mobile sensing affords this potential for context awareness in mobile computing, typically with a single sensing modality. Emerging multimodal large-scale language models offer new opportunities. We propose MultiSurf-GPT, which utilizes the advanced capabilities of GPT-4o to process and interpret diverse modalities (radar, microscope and multispectral data) uniformly based on prompting strategies (zero-shot and few-shot prompting). We preliminarily validated our framework by using MultiSurf-GPT to identify low-level information, and to infer high-level context-aware analytics, demonstrating the capability of augmenting context-aware insights. This framework shows promise as a tool to expedite the development of more complex context-aware applications in the future, providing a faster, more cost-effective, and integrated solution.<|reference_end|>
@article{hu2024multisurf-gpt:,
    title={MultiSurf-GPT: Facilitating Context-Aware Reasoning with Large-Scale
  Language Models for Multimodal Surface Sensing},
    author={Yongquan Hu, Black Sun, Pengcheng An, Zhuying Li, Wen Hu, Aaron J.
  Quigley},
    journal={arXiv preprint arXiv:2408.07311},
    year={2024},
    doi={10.1145/3640471.3680450},
    archivePrefix={arXiv},
    eprint={2408.07311},
    primaryClass={cs.HC}
}