title: SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction
    Fine-Tuning
summary: We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset
  designed for instruction fine-tuning and pre-training of speech-text large
  language models (LLMs). SIFT-50M is built from publicly available speech
  corpora, which collectively contain 14K hours of speech, and leverages LLMs
  along with off-the-shelf expert models. The dataset spans five languages,
  encompassing a diverse range of speech understanding as well as controllable
  speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which
  outperforms existing speech-text LLMs on instruction-following benchmarks while
  achieving competitive performance on foundational speech tasks. To support
  further research, we also introduce EvalSIFT, a benchmark dataset specifically
  designed to evaluate the instruction-following capabilities of speech-text
  LLMs.
paper_authors._id: 67fdbfcccaa65039e8c3d8ae
paper_authors.name: Prabhat Pandey
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:52.370Z
paper_authors._id: 67dcd93f73e2178fe917b893
paper_authors.avatarUrl: /avatars/86ec25d32c45da3937280ef9a619f19e.svg
paper_authors.fullname: Prabhat Pandey
paper_authors.type: user
paper_authors.user: panprabh
paper_authors._id: 67fdbfcccaa65039e8c3d8af
paper_authors.name: Rupak Vignesh Swaminathan
paper_authors._id: 67fdbfcccaa65039e8c3d8b0
paper_authors.name: K V Vijay Girish
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:49.809Z
paper_authors._id: 66279810009f1bfdc6bf71bf
paper_authors.avatarUrl: /avatars/97f561c91b92e1abb4fe6f0b5c688126.svg
paper_authors.fullname: Girish
paper_authors.type: user
paper_authors.user: vijaygirish2001
paper_authors._id: 67fdbfcccaa65039e8c3d8b1
paper_authors.name: Arunasish Sen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T11:58:01.767Z
paper_authors._id: 66367dfb2d6b86ff193dbbe0
paper_authors.avatarUrl: /avatars/0a0c230bb5fc81a28a166691146cf807.svg
paper_authors.fullname: Arunasish Sen
paper_authors.type: user
paper_authors.user: svinxz
paper_authors._id: 67fdbfcccaa65039e8c3d8b2
paper_authors.name: Jian Xie
paper_authors._id: 67fdbfcccaa65039e8c3d8b3
paper_authors.name: Grant P. Strimel
paper_authors._id: 67fdbfcccaa65039e8c3d8b4
paper_authors.name: Andreas Schwarz
paper_title: SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction
    Fine-Tuning
paper_summary: We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset
  designed for instruction fine-tuning and pre-training of speech-text large
  language models (LLMs). SIFT-50M is built from publicly available speech
  corpora, which collectively contain 14K hours of speech, and leverages LLMs
  along with off-the-shelf expert models. The dataset spans five languages,
  encompassing a diverse range of speech understanding as well as controllable
  speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which
  outperforms existing speech-text LLMs on instruction-following benchmarks while
  achieving competitive performance on foundational speech tasks. To support
  further research, we also introduce EvalSIFT, a benchmark dataset specifically
  designed to evaluate the instruction-following capabilities of speech-text
  LLMs.
paper_ai_summary: SIFT, a 50M-example speech-text dataset, fine-tunes and pre-trains LLMs for instruction-following and foundational speech tasks, outperforming existing models and using the EvalSIFT benchmark for evaluation.
paper_ai_keywords: SIFT instruction fine-tuning speech-text LLMs speech corpora LLMs SIFT-LLM EvalSIFT instruction-following benchmarks foundational speech tasks