Distilling an End-to-End Voice Assistant Without Instruction Training
  Data
Voice assistants, such as Siri and Google Assistant, typically model audio
and text separately, resulting in lost speech information and increased
complexity. Recent efforts to address this with end-to-end Speech Large
Language Models (LLMs) trained with supervised finetuning (SFT)
  have led to models ``forgetting" capabilities from text-only LLMs. Our work
proposes an alternative paradigm for training Speech LLMs without instruction
data, using the response of a text-only LLM to transcripts as self-supervision.
Importantly, this process can be performed without annotated responses. We show
that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question
Answering, Classification, and Translation. Furthermore, we show that DiVA
better meets user preferences, achieving a 72\% win rate compared with
state-of-the-art models like Qwen 2 Audio, despite using >100x less training
compute.
66ff775e7d722f087907e902
William Held
admin_assigned
2024-10-04T09:22:47.551Z
632116accafe12f481a473cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1666207676653-632116accafe12f481a473cb.jpeg
Will Held
user
WillHeld
66ff775e7d722f087907e903
Ella Li
claimed_verified
2024-11-05T07:58:50.780Z
632c0d9754e2c512c8f1550f
/avatars/223da1689dc81f790e4e5ce1b7bd5bb3.svg
Ella Li
user
Ella2323
66ff775e7d722f087907e904
Michael Ryan
claimed_verified
2024-10-04T20:11:23.640Z
63878fa2e40346f68ede7fc4
https://cdn-avatars.huggingface.co/v1/production/uploads/63878fa2e40346f68ede7fc4/MOdQbApvhwpVzPo5FWLVV.jpeg
Michael Ryan
user
MichaelR207
66ff775e7d722f087907e905
Weiyan Shi
admin_assigned
2024-10-04T09:23:55.783Z
65a37e6681a46e7dd96535d1
https://cdn-avatars.huggingface.co/v1/production/uploads/65a37e6681a46e7dd96535d1/hIpPK70k1FDE6iYOUtjQp.jpeg
Weiyan Shi
user
missckroach
66ff775e7d722f087907e906
Yanzhe Zhang
admin_assigned
2024-10-04T09:23:50.229Z
61362307df63bbd1c2459c4c
/avatars/28d3ba4b0fa00acb1eed623ab3620789.svg
Yanzhe Zhang
user
zyanzhe
66ff775e7d722f087907e907
Diyi Yang
Distilling an End-to-End Voice Assistant Without Instruction Training
  Data
Voice assistants, such as Siri and Google Assistant, typically model audio
and text separately, resulting in lost speech information and increased
complexity. Recent efforts to address this with end-to-end Speech Large
Language Models (LLMs) trained with supervised finetuning (SFT)
  have led to models ``forgetting" capabilities from text-only LLMs. Our work
proposes an alternative paradigm for training Speech LLMs without instruction
data, using the response of a text-only LLM to transcripts as self-supervision.
Importantly, this process can be performed without annotated responses. We show
that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question
Answering, Classification, and Translation. Furthermore, we show that DiVA
better meets user preferences, achieving a 72\% win rate compared with
state-of-the-art models like Qwen 2 Audio, despite using >100x less training
compute.
A self-supervised training approach using text-only LLM responses improves Speech LLMs for various tasks with reduced computational resources and enhanced user satisfaction.
Speech LLMs supervised finetuning (SFT) self-supervision Distilled Voice Assistant (DiVA) Spoken Question Answering classification translation