Soundwave: Less is More for Speech-Text Alignment in LLMs
Existing end-to-end speech large language models (LLMs) usually rely on
large-scale annotated data for training, while data-efficient training has not
been discussed in depth. We focus on two fundamental problems between speech
and text: the representation space gap and sequence length inconsistency. We
propose Soundwave, which utilizes an efficient training strategy and a novel
architecture to address these issues. Results show that Soundwave outperforms
the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks,
using only one-fiftieth of the training data. Further analysis shows that
Soundwave still retains its intelligence during conversation. The project is
available at https://github.com/FreedomIntelligence/Soundwave.
67b54851b986e35c41e063da
Yuhao Zhang
extracted_pending
2025-02-19T02:56:18.848Z
66975b9f8031bf92b428e138
/avatars/3254281a7bac1c8ddde1d6bc7e518b2f.svg
Yuhao Zhang
user
Yoohao
67b54851b986e35c41e063db
Zhiheng Liu
claimed_verified
2025-02-19T09:01:05.678Z
66597f2cf769c3c443b7cf41
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/WkZBh7hwlD9wVqCEMQtGX.png
Chihang Lau
user
puccho
67b54851b986e35c41e063dc
Fan Bu
admin_assigned
2025-02-19T09:42:08.544Z
668e7f46c243a12604035758
/avatars/35bd20032fafb7d7603266cf9a72d1e0.svg
Fan Bu
user
FanBuCUHK
67b54851b986e35c41e063dd
Ruiyu Zhang
admin_assigned
2025-02-19T09:42:14.866Z
67b587c8882e49771f610b51
/avatars/aecfb38b44141b8284416fc261692909.svg
Ruiyu Zhang
user
PhoenixAxis
67b54851b986e35c41e063de
Benyou Wang
admin_assigned
2025-02-19T09:42:23.845Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
67b54851b986e35c41e063df
Haizhou Li
Soundwave: Less is More for Speech-Text Alignment in LLMs
Existing end-to-end speech large language models (LLMs) usually rely on
large-scale annotated data for training, while data-efficient training has not
been discussed in depth. We focus on two fundamental problems between speech
and text: the representation space gap and sequence length inconsistency. We
propose Soundwave, which utilizes an efficient training strategy and a novel
architecture to address these issues. Results show that Soundwave outperforms
the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks,
using only one-fiftieth of the training data. Further analysis shows that
Soundwave still retains its intelligence during conversation. The project is
available at https://github.com/FreedomIntelligence/Soundwave.
Soundwave addresses the representation space gap and sequence length inconsistency in end-to-end speech large language models using an efficient training strategy and novel architecture, outperforming Qwen2-Audio with significantly less data.
large language models large-scale annotated data data-efficient training representation space gap sequence length inconsistency soundwave Qwen2-Audio speech translation AIR-Bench speech tasks