title: Distilling an End-to-End Voice Assistant Without Instruction Training
    Data
summary: Voice assistants, such as Siri and Google Assistant, typically model audio
  and text separately, resulting in lost speech information and increased
  complexity. Recent efforts to address this with end-to-end Speech Large
  Language Models (LLMs) trained with supervised finetuning (SFT)
    have led to models ``forgetting" capabilities from text-only LLMs. Our work
  proposes an alternative paradigm for training Speech LLMs without instruction
  data, using the response of a text-only LLM to transcripts as self-supervision.
  Importantly, this process can be performed without annotated responses. We show
  that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question
  Answering, Classification, and Translation. Furthermore, we show that DiVA
  better meets user preferences, achieving a 72\% win rate compared with
  state-of-the-art models like Qwen 2 Audio, despite using >100x less training
  compute.
paper_authors._id: 66ff775e7d722f087907e902
paper_authors.name: William Held
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:22:47.551Z
paper_authors._id: 632116accafe12f481a473cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666207676653-632116accafe12f481a473cb.jpeg
paper_authors.fullname: Will Held
paper_authors.type: user
paper_authors.user: WillHeld
paper_authors._id: 66ff775e7d722f087907e903
paper_authors.name: Ella Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:58:50.780Z
paper_authors._id: 632c0d9754e2c512c8f1550f
paper_authors.avatarUrl: /avatars/223da1689dc81f790e4e5ce1b7bd5bb3.svg
paper_authors.fullname: Ella Li
paper_authors.type: user
paper_authors.user: Ella2323
paper_authors._id: 66ff775e7d722f087907e904
paper_authors.name: Michael Ryan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T20:11:23.640Z
paper_authors._id: 63878fa2e40346f68ede7fc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63878fa2e40346f68ede7fc4/MOdQbApvhwpVzPo5FWLVV.jpeg
paper_authors.fullname: Michael Ryan
paper_authors.type: user
paper_authors.user: MichaelR207
paper_authors._id: 66ff775e7d722f087907e905
paper_authors.name: Weiyan Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:23:55.783Z
paper_authors._id: 65a37e6681a46e7dd96535d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a37e6681a46e7dd96535d1/hIpPK70k1FDE6iYOUtjQp.jpeg
paper_authors.fullname: Weiyan Shi
paper_authors.type: user
paper_authors.user: missckroach
paper_authors._id: 66ff775e7d722f087907e906
paper_authors.name: Yanzhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:23:50.229Z
paper_authors._id: 61362307df63bbd1c2459c4c
paper_authors.avatarUrl: /avatars/28d3ba4b0fa00acb1eed623ab3620789.svg
paper_authors.fullname: Yanzhe Zhang
paper_authors.type: user
paper_authors.user: zyanzhe
paper_authors._id: 66ff775e7d722f087907e907
paper_authors.name: Diyi Yang
paper_title: Distilling an End-to-End Voice Assistant Without Instruction Training
    Data
paper_summary: Voice assistants, such as Siri and Google Assistant, typically model audio
  and text separately, resulting in lost speech information and increased
  complexity. Recent efforts to address this with end-to-end Speech Large
  Language Models (LLMs) trained with supervised finetuning (SFT)
    have led to models ``forgetting" capabilities from text-only LLMs. Our work
  proposes an alternative paradigm for training Speech LLMs without instruction
  data, using the response of a text-only LLM to transcripts as self-supervision.
  Importantly, this process can be performed without annotated responses. We show
  that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question
  Answering, Classification, and Translation. Furthermore, we show that DiVA
  better meets user preferences, achieving a 72\% win rate compared with
  state-of-the-art models like Qwen 2 Audio, despite using >100x less training
  compute.
paper_ai_summary: A self-supervised training approach using text-only LLM responses improves Speech LLMs for various tasks with reduced computational resources and enhanced user satisfaction.
paper_ai_keywords: Speech LLMs supervised finetuning (SFT) self-supervision Distilled Voice Assistant (DiVA) Spoken Question Answering classification translation