title: Multimodal Data and Resource Efficient Device-Directed Speech Detection
    with Large Foundation Models
summary: Interactions with virtual assistants typically start with a trigger phrase
  followed by a command. In this work, we explore the possibility of making these
  interactions more natural by eliminating the need for a trigger phrase. Our
  goal is to determine whether a user addressed the virtual assistant based on
  signals obtained from the streaming audio recorded by the device microphone. We
  address this task by combining 1-best hypotheses and decoder signals from an
  automatic speech recognition system with acoustic representations from an audio
  encoder as input features to a large language model (LLM). In particular, we
  are interested in data and resource efficient systems that require only a small
  amount of training data and can operate in scenarios with only a single frozen
  LLM available on a device. For this reason, our model is trained on 80k or less
  examples of multimodal data using a combination of low-rank adaptation and
  prefix tuning. We compare the proposed system to unimodal baselines and show
  that the multimodal approach achieves lower equal-error-rates (EERs), while
  using only a fraction of the training data. We also show that low-dimensional
  specialized audio representations lead to lower EERs than high-dimensional
  general audio representations.
paper_authors._id: 6571275f0ea91e592a1cebb5
paper_authors.name: Dominik Wagner
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T15:10:41.032Z
paper_authors._id: 6311ba6f05cc08a1408d910a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662997515866-6311ba6f05cc08a1408d910a.png
paper_authors.fullname: Dominik Wagner
paper_authors.type: user
paper_authors.user: dwgnr
paper_authors._id: 6571275f0ea91e592a1cebb6
paper_authors.name: Alexander Churchill
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:23:22.260Z
paper_authors._id: 630fd935dd31b2a8dbe70abf
paper_authors.avatarUrl: /avatars/49711a6644a1b8fb13670ac06af87693.svg
paper_authors.fullname: Alex Churchill
paper_authors.type: user
paper_authors.user: Alexanderchurchill
paper_authors._id: 6571275f0ea91e592a1cebb7
paper_authors.name: Siddharth Sigtia
paper_authors._id: 6571275f0ea91e592a1cebb8
paper_authors.name: Panayiotis Georgiou
paper_authors._id: 6571275f0ea91e592a1cebb9
paper_authors.name: Matt Mirsamadi
paper_authors._id: 6571275f0ea91e592a1cebba
paper_authors.name: Aarshee Mishra
paper_authors._id: 6571275f0ea91e592a1cebbb
paper_authors.name: Erik Marchi
paper_title: Multimodal Data and Resource Efficient Device-Directed Speech Detection
    with Large Foundation Models
paper_summary: Interactions with virtual assistants typically start with a trigger phrase
  followed by a command. In this work, we explore the possibility of making these
  interactions more natural by eliminating the need for a trigger phrase. Our
  goal is to determine whether a user addressed the virtual assistant based on
  signals obtained from the streaming audio recorded by the device microphone. We
  address this task by combining 1-best hypotheses and decoder signals from an
  automatic speech recognition system with acoustic representations from an audio
  encoder as input features to a large language model (LLM). In particular, we
  are interested in data and resource efficient systems that require only a small
  amount of training data and can operate in scenarios with only a single frozen
  LLM available on a device. For this reason, our model is trained on 80k or less
  examples of multimodal data using a combination of low-rank adaptation and
  prefix tuning. We compare the proposed system to unimodal baselines and show
  that the multimodal approach achieves lower equal-error-rates (EERs), while
  using only a fraction of the training data. We also show that low-dimensional
  specialized audio representations lead to lower EERs than high-dimensional
  general audio representations.
paper_ai_summary: A multimodal system combining ASR and audio encoding improves natural interaction with virtual assistants by reducing the need for trigger phrases and using efficient learning techniques.
paper_ai_keywords: automatic speech recognition audio encoder large language model low-rank adaptation prefix tuning equal-error-rate