title: Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based
    Speech Synthesis
summary: Recent advances in text-based large language models (LLMs), particularly in
  the GPT series and the o1 model, have demonstrated the effectiveness of scaling
  both training-time and inference-time compute. However, current
  state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring
  separate models (e.g., diffusion models after LLM), complicating the decision
  of whether to scale a particular model during training or testing. This work
  makes the following contributions: First, we explore the scaling of train-time
  and inference-time compute for speech synthesis. Second, we propose a simple
  framework Llasa for speech synthesis that employs a single-layer vector
  quantizer (VQ) codec and a single Transformer architecture to fully align with
  standard LLMs such as Llama. Our experiments reveal that scaling train-time
  compute for Llasa consistently improves the naturalness of synthesized speech
  and enables the generation of more complex and accurate prosody patterns.
  Furthermore, from the perspective of scaling inference-time compute, we employ
  speech understanding models as verifiers during the search, finding that
  scaling inference-time compute shifts the sampling modes toward the preferences
  of specific verifiers, thereby improving emotional expressiveness, timbre
  consistency, and content accuracy. In addition, we released the checkpoint and
  training code for our TTS model (1B, 3B, 8B) and codec model publicly
  available.
paper_authors._id: 67a5894db16fabcdd2de5459
paper_authors.name: Zhen Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T09:58:08.787Z
paper_authors._id: 645f172d7c6bff8577353d1a
paper_authors.avatarUrl: /avatars/a83682e1343809257b082b78d58c582a.svg
paper_authors.fullname: ZhenYE
paper_authors.type: user
paper_authors.user: ZhenYe234
paper_authors._id: 67a5894db16fabcdd2de545a
paper_authors.name: Xinfa Zhu
paper_authors._id: 67a5894db16fabcdd2de545b
paper_authors.name: Chi-Min Chan
paper_authors._id: 67a5894db16fabcdd2de545c
paper_authors.name: Xinsheng Wang
paper_authors._id: 67a5894db16fabcdd2de545d
paper_authors.name: Xu Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:45:27.312Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 67a5894db16fabcdd2de545e
paper_authors.name: Jiahe Lei
paper_authors._id: 67a5894db16fabcdd2de545f
paper_authors.name: Yi Peng
paper_authors._id: 67a5894db16fabcdd2de5460
paper_authors.name: Haohe Liu
paper_authors._id: 67a5894db16fabcdd2de5461
paper_authors.name: Yizhu Jin
paper_authors._id: 67a5894db16fabcdd2de5462
paper_authors.name: Zheqi DAI
paper_authors._id: 67a5894db16fabcdd2de5463
paper_authors.name: Hongzhan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:05:09.037Z
paper_authors._id: 6499466c7d1edf7cb612a9a6
paper_authors.avatarUrl: /avatars/c2e18594aa0879db8226f2a04496fb0b.svg
paper_authors.fullname: Hongzhan Lin
paper_authors.type: user
paper_authors.user: danielhzlin
paper_authors._id: 67a5894db16fabcdd2de5464
paper_authors.name: Jianyi Chen
paper_authors._id: 67a5894db16fabcdd2de5465
paper_authors.name: Xingjian Du
paper_authors._id: 67a5894db16fabcdd2de5466
paper_authors.name: Liumeng Xue
paper_authors._id: 67a5894db16fabcdd2de5467
paper_authors.name: Yunlin Chen
paper_authors._id: 67a5894db16fabcdd2de5468
paper_authors.name: Zhifei Li
paper_authors._id: 67a5894db16fabcdd2de5469
paper_authors.name: Lei Xie
paper_authors._id: 67a5894db16fabcdd2de546a
paper_authors.name: Qiuqiang Kong
paper_authors._id: 67a5894db16fabcdd2de546b
paper_authors.name: Yike Guo
paper_authors._id: 67a5894db16fabcdd2de546c
paper_authors.name: Wei Xue
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-07T04:17:17.888Z
paper_authors._id: 6628adb14277eae0da5eee28
paper_authors.avatarUrl: /avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
paper_authors.fullname: HKUST Audio
paper_authors.type: user
paper_authors.user: HKUST-Audio
paper_title: Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based
    Speech Synthesis
paper_summary: Recent advances in text-based large language models (LLMs), particularly in
  the GPT series and the o1 model, have demonstrated the effectiveness of scaling
  both training-time and inference-time compute. However, current
  state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring
  separate models (e.g., diffusion models after LLM), complicating the decision
  of whether to scale a particular model during training or testing. This work
  makes the following contributions: First, we explore the scaling of train-time
  and inference-time compute for speech synthesis. Second, we propose a simple
  framework Llasa for speech synthesis that employs a single-layer vector
  quantizer (VQ) codec and a single Transformer architecture to fully align with
  standard LLMs such as Llama. Our experiments reveal that scaling train-time
  compute for Llasa consistently improves the naturalness of synthesized speech
  and enables the generation of more complex and accurate prosody patterns.
  Furthermore, from the perspective of scaling inference-time compute, we employ
  speech understanding models as verifiers during the search, finding that
  scaling inference-time compute shifts the sampling modes toward the preferences
  of specific verifiers, thereby improving emotional expressiveness, timbre
  consistency, and content accuracy. In addition, we released the checkpoint and
  training code for our TTS model (1B, 3B, 8B) and codec model publicly
  available.
paper_ai_summary: A single-layer vector quantizer codec and Transformer-based framework for text-to-speech synthesis improves speech naturalness and emotional expressiveness with scalable compute resources.
paper_ai_keywords: large language models GPT series diffusion models speech synthesis vector quantizer (VQ) codec Transformer architecture train-time compute inference-time compute prosody patterns speech understanding models verifiers sampling modes emotional expressiveness timbre consistency content accuracy