title: AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
summary: Auditory foundation models, including auditory large language models (LLMs),
  process all sound inputs equally, independent of listener perception. However,
  human auditory perception is inherently selective: listeners focus on specific
  speakers while ignoring others in complex auditory scenes. Existing models do
  not incorporate this selectivity, limiting their ability to generate
  perception-aligned responses. To address this, we introduce Intention-Informed
  Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM
  (AAD-LLM), a prototype system that integrates brain signals to infer listener
  attention. AAD-LLM extends an auditory LLM by incorporating intracranial
  electroencephalography (iEEG) recordings to decode which speaker a listener is
  attending to and refine responses accordingly. The model first predicts the
  attended speaker from neural activity, then conditions response generation on
  this inferred attentional state. We evaluate AAD-LLM on speaker description,
  speech transcription and extraction, and question answering in multitalker
  scenarios, with both objective and subjective ratings showing improved
  alignment with listener intention. By taking a first step toward
  intention-aware auditory AI, this work explores a new paradigm where listener
  perception informs machine listening, paving the way for future
  listener-centered auditory systems. Demo and code available:
  https://aad-llm.github.io.
paper_authors._id: 67be86a78a5a80542314f0e6
paper_authors.name: Xilin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:25:52.841Z
paper_authors._id: 6531a65daed617662c7f1007
paper_authors.avatarUrl: /avatars/ea2e504780dc40719f7501ab2c7d9c91.svg
paper_authors.fullname: Xilin Jiang
paper_authors.type: user
paper_authors.user: xi-j
paper_authors._id: 67be86a78a5a80542314f0e7
paper_authors.name: Sukru Samet Dindar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T09:11:37.706Z
paper_authors._id: 661361993bb67cb4f356c3de
paper_authors.avatarUrl: /avatars/b707c07f9c70d2ed1e8cd8cff2551c69.svg
paper_authors.fullname: Sukru Samet Dindar
paper_authors.type: user
paper_authors.user: susameddin
paper_authors._id: 67be86a78a5a80542314f0e8
paper_authors.name: Vishal Choudhari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T09:11:45.258Z
paper_authors._id: 670e8671ba29b3fca221b8c9
paper_authors.avatarUrl: /avatars/20f6479bd5218d6d3e304539df5003f9.svg
paper_authors.fullname: Vishal Choudhari
paper_authors.type: user
paper_authors.user: vchoudhari
paper_authors._id: 67be86a78a5a80542314f0e9
paper_authors.name: Stephan Bickel
paper_authors._id: 67be86a78a5a80542314f0ea
paper_authors.name: Ashesh Mehta
paper_authors._id: 67be86a78a5a80542314f0eb
paper_authors.name: Guy M McKhann
paper_authors._id: 67be86a78a5a80542314f0ec
paper_authors.name: Adeen Flinker
paper_authors._id: 67be86a78a5a80542314f0ed
paper_authors.name: Daniel Friedman
paper_authors._id: 67be86a78a5a80542314f0ee
paper_authors.name: Nima Mesgarani
paper_title: AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
paper_summary: Auditory foundation models, including auditory large language models (LLMs),
  process all sound inputs equally, independent of listener perception. However,
  human auditory perception is inherently selective: listeners focus on specific
  speakers while ignoring others in complex auditory scenes. Existing models do
  not incorporate this selectivity, limiting their ability to generate
  perception-aligned responses. To address this, we introduce Intention-Informed
  Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM
  (AAD-LLM), a prototype system that integrates brain signals to infer listener
  attention. AAD-LLM extends an auditory LLM by incorporating intracranial
  electroencephalography (iEEG) recordings to decode which speaker a listener is
  attending to and refine responses accordingly. The model first predicts the
  attended speaker from neural activity, then conditions response generation on
  this inferred attentional state. We evaluate AAD-LLM on speaker description,
  speech transcription and extraction, and question answering in multitalker
  scenarios, with both objective and subjective ratings showing improved
  alignment with listener intention. By taking a first step toward
  intention-aware auditory AI, this work explores a new paradigm where listener
  perception informs machine listening, paving the way for future
  listener-centered auditory systems. Demo and code available:
  https://aad-llm.github.io.
paper_ai_keywords: Auditory foundation models auditory large language models (LLMs) auditory perception human auditory perception auditory scene understanding (ASU) Intention-Informed Auditory Scene Understanding (II-ASU) Auditory Attention-Driven LLM (AAD-LLM) brain signals intracranial electroencephalography (iEEG) neural activity response generation speaker description speech transcription question answering multitalker scenarios