AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Auditory foundation models, including auditory large language models (LLMs),
process all sound inputs equally, independent of listener perception. However,
human auditory perception is inherently selective: listeners focus on specific
speakers while ignoring others in complex auditory scenes. Existing models do
not incorporate this selectivity, limiting their ability to generate
perception-aligned responses. To address this, we introduce Intention-Informed
Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM
(AAD-LLM), a prototype system that integrates brain signals to infer listener
attention. AAD-LLM extends an auditory LLM by incorporating intracranial
electroencephalography (iEEG) recordings to decode which speaker a listener is
attending to and refine responses accordingly. The model first predicts the
attended speaker from neural activity, then conditions response generation on
this inferred attentional state. We evaluate AAD-LLM on speaker description,
speech transcription and extraction, and question answering in multitalker
scenarios, with both objective and subjective ratings showing improved
alignment with listener intention. By taking a first step toward
intention-aware auditory AI, this work explores a new paradigm where listener
perception informs machine listening, paving the way for future
listener-centered auditory systems. Demo and code available:
https://aad-llm.github.io.
67be86a78a5a80542314f0e6
Xilin Jiang
claimed_verified
2025-02-26T08:25:52.841Z
6531a65daed617662c7f1007
/avatars/ea2e504780dc40719f7501ab2c7d9c91.svg
Xilin Jiang
user
xi-j
67be86a78a5a80542314f0e7
Sukru Samet Dindar
admin_assigned
2025-02-26T09:11:37.706Z
661361993bb67cb4f356c3de
/avatars/b707c07f9c70d2ed1e8cd8cff2551c69.svg
Sukru Samet Dindar
user
susameddin
67be86a78a5a80542314f0e8
Vishal Choudhari
admin_assigned
2025-02-26T09:11:45.258Z
670e8671ba29b3fca221b8c9
/avatars/20f6479bd5218d6d3e304539df5003f9.svg
Vishal Choudhari
user
vchoudhari
67be86a78a5a80542314f0e9
Stephan Bickel
67be86a78a5a80542314f0ea
Ashesh Mehta
67be86a78a5a80542314f0eb
Guy M McKhann
67be86a78a5a80542314f0ec
Adeen Flinker
67be86a78a5a80542314f0ed
Daniel Friedman
67be86a78a5a80542314f0ee
Nima Mesgarani
AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Auditory foundation models, including auditory large language models (LLMs),
process all sound inputs equally, independent of listener perception. However,
human auditory perception is inherently selective: listeners focus on specific
speakers while ignoring others in complex auditory scenes. Existing models do
not incorporate this selectivity, limiting their ability to generate
perception-aligned responses. To address this, we introduce Intention-Informed
Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM
(AAD-LLM), a prototype system that integrates brain signals to infer listener
attention. AAD-LLM extends an auditory LLM by incorporating intracranial
electroencephalography (iEEG) recordings to decode which speaker a listener is
attending to and refine responses accordingly. The model first predicts the
attended speaker from neural activity, then conditions response generation on
this inferred attentional state. We evaluate AAD-LLM on speaker description,
speech transcription and extraction, and question answering in multitalker
scenarios, with both objective and subjective ratings showing improved
alignment with listener intention. By taking a first step toward
intention-aware auditory AI, this work explores a new paradigm where listener
perception informs machine listening, paving the way for future
listener-centered auditory systems. Demo and code available:
https://aad-llm.github.io.
AAD-LLM integrates neural signals to infer listener attention, improving auditory LLM responses in complex scenes by aligning with listener intentions.
auditory large language models auditory perception selectivity listener attention intention-informed auditory scene understanding Auditory Attention-Driven LLM intracranial electroencephalography neural activity speaker description speech transcription question answering multitalker scenarios listener-centered