title: Soundwave: Less is More for Speech-Text Alignment in LLMs
summary: Existing end-to-end speech large language models (LLMs) usually rely on
  large-scale annotated data for training, while data-efficient training has not
  been discussed in depth. We focus on two fundamental problems between speech
  and text: the representation space gap and sequence length inconsistency. We
  propose Soundwave, which utilizes an efficient training strategy and a novel
  architecture to address these issues. Results show that Soundwave outperforms
  the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks,
  using only one-fiftieth of the training data. Further analysis shows that
  Soundwave still retains its intelligence during conversation. The project is
  available at https://github.com/FreedomIntelligence/Soundwave.
paper_authors._id: 67b54851b986e35c41e063da
paper_authors.name: Yuhao Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-19T02:56:18.848Z
paper_authors._id: 66975b9f8031bf92b428e138
paper_authors.avatarUrl: /avatars/3254281a7bac1c8ddde1d6bc7e518b2f.svg
paper_authors.fullname: Yuhao Zhang
paper_authors.type: user
paper_authors.user: Yoohao
paper_authors._id: 67b54851b986e35c41e063db
paper_authors.name: Zhiheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:01:05.678Z
paper_authors._id: 66597f2cf769c3c443b7cf41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/WkZBh7hwlD9wVqCEMQtGX.png
paper_authors.fullname: Chihang Lau
paper_authors.type: user
paper_authors.user: puccho
paper_authors._id: 67b54851b986e35c41e063dc
paper_authors.name: Fan Bu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:42:08.544Z
paper_authors._id: 668e7f46c243a12604035758
paper_authors.avatarUrl: /avatars/35bd20032fafb7d7603266cf9a72d1e0.svg
paper_authors.fullname: Fan Bu
paper_authors.type: user
paper_authors.user: FanBuCUHK
paper_authors._id: 67b54851b986e35c41e063dd
paper_authors.name: Ruiyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:42:14.866Z
paper_authors._id: 67b587c8882e49771f610b51
paper_authors.avatarUrl: /avatars/aecfb38b44141b8284416fc261692909.svg
paper_authors.fullname: Ruiyu Zhang
paper_authors.type: user
paper_authors.user: PhoenixAxis
paper_authors._id: 67b54851b986e35c41e063de
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:42:23.845Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 67b54851b986e35c41e063df
paper_authors.name: Haizhou Li
paper_title: Soundwave: Less is More for Speech-Text Alignment in LLMs
paper_summary: Existing end-to-end speech large language models (LLMs) usually rely on
  large-scale annotated data for training, while data-efficient training has not
  been discussed in depth. We focus on two fundamental problems between speech
  and text: the representation space gap and sequence length inconsistency. We
  propose Soundwave, which utilizes an efficient training strategy and a novel
  architecture to address these issues. Results show that Soundwave outperforms
  the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks,
  using only one-fiftieth of the training data. Further analysis shows that
  Soundwave still retains its intelligence during conversation. The project is
  available at https://github.com/FreedomIntelligence/Soundwave.
paper_ai_summary: Soundwave addresses the representation space gap and sequence length inconsistency in end-to-end speech large language models using an efficient training strategy and novel architecture, outperforming Qwen2-Audio with significantly less data.
paper_ai_keywords: large language models large-scale annotated data data-efficient training representation space gap sequence length inconsistency soundwave Qwen2-Audio speech translation AIR-Bench speech tasks