Investigating Decoder-only Large Language Models for Speech-to-text
  Translation
Large language models (LLMs), known for their exceptional reasoning
capabilities, generalizability, and fluency across diverse domains, present a
promising avenue for enhancing speech-related tasks. In this paper, we focus on
integrating decoder-only LLMs to the task of speech-to-text translation (S2TT).
We propose a decoder-only architecture that enables the LLM to directly consume
the encoded speech representation and generate the text translation.
Additionally, we investigate the effects of different parameter-efficient
fine-tuning techniques and task formulation. Our model achieves
state-of-the-art performance on CoVoST 2 and FLEURS among models trained
without proprietary data. We also conduct analyses to validate the design
choices of our proposed model and bring insights to the integration of LLMs to
S2TT.
6685fe7fb615543f13b11a23
Chao-Wei Huang
claimed_verified
2024-07-04T08:03:44.624Z
5e060e7dcbfd036a99df0dbf
https://cdn-avatars.huggingface.co/v1/production/uploads/5e060e7dcbfd036a99df0dbf/qJ31KnXhyP701B2vcot8Q.jpeg
Chao-Wei Huang
user
chaoweihuang
6685fe7fb615543f13b11a24
Hui Lu
admin_assigned
2024-07-04T12:47:49.628Z
648130953fb124fc984fe582
/avatars/2334eff6fec16fdc551f453f6e363df7.svg
Hui Lu
user
huilu
6685fe7fb615543f13b11a25
Hongyu Gong
admin_assigned
2024-07-04T12:47:58.118Z
634dedb9e2d037c1280229ed
/avatars/1a4bb70ca3a5ed1a6d201324d0de9858.svg
Hongyu Gong
user
hygong
6685fe7fb615543f13b11a26
Hirofumi Inaguma
6685fe7fb615543f13b11a27
Ilia Kulikov
admin_assigned
2024-07-04T12:48:11.445Z
65d645913bc37244c55c39d5
/avatars/508372876a654957650ee9429886d10c.svg
ilia kulikov
user
uralik
6685fe7fb615543f13b11a28
Ruslan Mavlyutov
admin_assigned
2024-07-04T12:48:17.106Z
64dcfecb249785efad9a514d
/avatars/77de68494021f298bf2561b2ea9ffef8.svg
Ruslan Mavlyutov
user
mavlyutovrus
6685fe7fb615543f13b11a29
Sravya Popuri
admin_assigned
2024-07-04T12:48:23.311Z
62e190f3926f4892a4c8099e
/avatars/2168f769c526362f9ed35473f4b30047.svg
Sravya Popuri
user
sravyapopuri388
Investigating Decoder-only Large Language Models for Speech-to-text
  Translation
Large language models (LLMs), known for their exceptional reasoning
capabilities, generalizability, and fluency across diverse domains, present a
promising avenue for enhancing speech-related tasks. In this paper, we focus on
integrating decoder-only LLMs to the task of speech-to-text translation (S2TT).
We propose a decoder-only architecture that enables the LLM to directly consume
the encoded speech representation and generate the text translation.
Additionally, we investigate the effects of different parameter-efficient
fine-tuning techniques and task formulation. Our model achieves
state-of-the-art performance on CoVoST 2 and FLEURS among models trained
without proprietary data. We also conduct analyses to validate the design
choices of our proposed model and bring insights to the integration of LLMs to
S2TT.
Decoder-only large language models integrated with speech-to-text translation achieve state-of-the-art performance without proprietary data through parameter-efficient fine-tuning.
decoder-only LLMs speech-to-text translation CoVoST 2 FLEURS parameter-efficient fine-tuning