IoT-LM: Large Multisensory Language Models for the Internet of Things
<|reference_start|>IoT-LM: Large Multisensory Language Models for the Internet of Things: The Internet of Things (IoT) network integrating billions of smart physical devices embedded with sensors, software, and communication technologies is a critical and rapidly expanding component of our modern world. The IoT ecosystem provides a rich source of real-world modalities such as motion, thermal, geolocation, imaging, depth, sensors, and audio to recognize the states of humans and physical objects. Machine learning presents a rich opportunity to automatically process IoT data at scale, enabling efficient inference for understanding human wellbeing, controlling physical devices, and interconnecting smart cities. To realize this potential, we introduce IoT-LM, an open-source large multisensory language model tailored for the IoT ecosystem. IoT-LM is enabled by two technical contributions: the first is MultiIoT, the most expansive unified IoT dataset to date, encompassing over 1.15 million samples from 12 modalities and 8 tasks prepared for multisensory pre-training and instruction-tuning. The second is a new multisensory multitask adapter layer to condition pre-trained large language models on multisensory IoT data. Not only does IoT-LM yield substantial improvements on 8 supervised IoT classification tasks, but it also demonstrates new interactive question-answering, reasoning, and dialog capabilities conditioned on IoT sensors. We release IoT-LM's data sources and new multisensory language modeling framework.<|reference_end|>
@article{mo2024iot-lm:,
    title={IoT-LM: Large Multisensory Language Models for the Internet of Things},
    author={Shentong Mo, Russ Salakhutdinov, Louis-Philippe Morency, Paul Pu Liang},
    journal={arXiv preprint arXiv:2407.09801},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.09801},
    primaryClass={cs.LG cs.AI cs.CL cs.CV cs.MM}
}