Can Large Language Models Understand Spatial Audio?
<|reference_start|>Can Large Language Models Understand Spatial Audio?: This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMs' advanced cognitive and inferential abilities, the aim is to enhance understanding of 3D environments via audio. We study 3 spatial audio tasks: sound source localization (SSL), far-field speech recognition (FSR), and localisation-informed speech extraction (LSE), achieving notable progress in each task. For SSL, our approach achieves an MAE of $2.70^{\circ}$ on the Spatial LibriSpeech dataset, substantially surpassing the prior benchmark of about $6.60^{\circ}$. Moreover, our model can employ spatial cues to improve FSR accuracy and execute LSE by selectively attending to sounds originating from a specified direction via text prompts, even amidst overlapping speech. These findings highlight the potential of adapting LLMs to grasp physical audio concepts, paving the way for LLM-based agents in 3D environments.<|reference_end|>
@article{tang2024can,
    title={Can Large Language Models Understand Spatial Audio?},
    author={Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li,
  Jun Zhang, Lu Lu, Zejun Ma, Yuxuan Wang, Chao Zhang},
    journal={arXiv preprint arXiv:2406.07914},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.07914},
    primaryClass={cs.SD eess.AS}
}