title: Investigating Decoder-only Large Language Models for Speech-to-text
    Translation
summary: Large language models (LLMs), known for their exceptional reasoning
  capabilities, generalizability, and fluency across diverse domains, present a
  promising avenue for enhancing speech-related tasks. In this paper, we focus on
  integrating decoder-only LLMs to the task of speech-to-text translation (S2TT).
  We propose a decoder-only architecture that enables the LLM to directly consume
  the encoded speech representation and generate the text translation.
  Additionally, we investigate the effects of different parameter-efficient
  fine-tuning techniques and task formulation. Our model achieves
  state-of-the-art performance on CoVoST 2 and FLEURS among models trained
  without proprietary data. We also conduct analyses to validate the design
  choices of our proposed model and bring insights to the integration of LLMs to
  S2TT.
paper_authors._id: 6685fe7fb615543f13b11a23
paper_authors.name: Chao-Wei Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T08:03:44.624Z
paper_authors._id: 5e060e7dcbfd036a99df0dbf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e060e7dcbfd036a99df0dbf/qJ31KnXhyP701B2vcot8Q.jpeg
paper_authors.fullname: Chao-Wei Huang
paper_authors.type: user
paper_authors.user: chaoweihuang
paper_authors._id: 6685fe7fb615543f13b11a24
paper_authors.name: Hui Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:47:49.628Z
paper_authors._id: 648130953fb124fc984fe582
paper_authors.avatarUrl: /avatars/2334eff6fec16fdc551f453f6e363df7.svg
paper_authors.fullname: Hui Lu
paper_authors.type: user
paper_authors.user: huilu
paper_authors._id: 6685fe7fb615543f13b11a25
paper_authors.name: Hongyu Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:47:58.118Z
paper_authors._id: 634dedb9e2d037c1280229ed
paper_authors.avatarUrl: /avatars/1a4bb70ca3a5ed1a6d201324d0de9858.svg
paper_authors.fullname: Hongyu Gong
paper_authors.type: user
paper_authors.user: hygong
paper_authors._id: 6685fe7fb615543f13b11a26
paper_authors.name: Hirofumi Inaguma
paper_authors._id: 6685fe7fb615543f13b11a27
paper_authors.name: Ilia Kulikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:48:11.445Z
paper_authors._id: 65d645913bc37244c55c39d5
paper_authors.avatarUrl: /avatars/508372876a654957650ee9429886d10c.svg
paper_authors.fullname: ilia kulikov
paper_authors.type: user
paper_authors.user: uralik
paper_authors._id: 6685fe7fb615543f13b11a28
paper_authors.name: Ruslan Mavlyutov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:48:17.106Z
paper_authors._id: 64dcfecb249785efad9a514d
paper_authors.avatarUrl: /avatars/77de68494021f298bf2561b2ea9ffef8.svg
paper_authors.fullname: Ruslan Mavlyutov
paper_authors.type: user
paper_authors.user: mavlyutovrus
paper_authors._id: 6685fe7fb615543f13b11a29
paper_authors.name: Sravya Popuri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:48:23.311Z
paper_authors._id: 62e190f3926f4892a4c8099e
paper_authors.avatarUrl: /avatars/2168f769c526362f9ed35473f4b30047.svg
paper_authors.fullname: Sravya Popuri
paper_authors.type: user
paper_authors.user: sravyapopuri388
paper_title: Investigating Decoder-only Large Language Models for Speech-to-text
    Translation
paper_summary: Large language models (LLMs), known for their exceptional reasoning
  capabilities, generalizability, and fluency across diverse domains, present a
  promising avenue for enhancing speech-related tasks. In this paper, we focus on
  integrating decoder-only LLMs to the task of speech-to-text translation (S2TT).
  We propose a decoder-only architecture that enables the LLM to directly consume
  the encoded speech representation and generate the text translation.
  Additionally, we investigate the effects of different parameter-efficient
  fine-tuning techniques and task formulation. Our model achieves
  state-of-the-art performance on CoVoST 2 and FLEURS among models trained
  without proprietary data. We also conduct analyses to validate the design
  choices of our proposed model and bring insights to the integration of LLMs to
  S2TT.
paper_ai_summary: Decoder-only large language models integrated with speech-to-text translation achieve state-of-the-art performance without proprietary data through parameter-efficient fine-tuning.
paper_ai_keywords: decoder-only LLMs speech-to-text translation CoVoST 2 FLEURS parameter-efficient fine-tuning