title: Multimodal Latent Language Modeling with Next-Token Diffusion
summary: Multimodal generative models require a unified approach to handle both
  discrete data (e.g., text and code) and continuous data (e.g., image, audio,
  video). In this work, we propose Latent Language Modeling (LatentLM), which
  seamlessly integrates continuous and discrete data using causal Transformers.
  Specifically, we employ a variational autoencoder (VAE) to represent continuous
  data as latent vectors and introduce next-token diffusion for autoregressive
  generation of these vectors. Additionally, we develop sigma-VAE to address
  the challenges of variance collapse, which is crucial for autoregressive
  modeling. Extensive experiments demonstrate the effectiveness of LatentLM
  across various modalities. In image generation, LatentLM surpasses Diffusion
  Transformers in both performance and scalability. When integrated into
  multimodal large language models, LatentLM provides a general-purpose interface
  that unifies multimodal generation and understanding. Experimental results show
  that LatentLM achieves favorable performance compared to Transfusion and vector
  quantized models in the setting of scaling up training tokens. In
  text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2
  model in speaker similarity and robustness, while requiring 10x fewer decoding
  steps. The results establish LatentLM as a highly effective and scalable
  approach to advance large multimodal models.
paper_authors._id: 675b17aa378af168bbfd66fa
paper_authors.name: Yutao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:41:50.828Z
paper_authors._id: 6300ef4779c5ddbc6cf83e1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661005591657-noauth.jpeg
paper_authors.fullname: Yutao Sun
paper_authors.type: user
paper_authors.user: sunyt32
paper_authors._id: 675b17aa378af168bbfd66fb
paper_authors.name: Hangbo Bao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:41:44.329Z
paper_authors._id: 62c6981c9db11473f08b0c87
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c6981c9db11473f08b0c87/PKnnQ1SOrMuB4vf2cDDdG.jpeg
paper_authors.fullname: Hangbo Bao
paper_authors.type: user
paper_authors.user: addf400
paper_authors._id: 675b17aa378af168bbfd66fc
paper_authors.name: Wenhui Wang
paper_authors._id: 675b17aa378af168bbfd66fd
paper_authors.name: Zhiliang Peng
paper_authors._id: 675b17aa378af168bbfd66fe
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:23:05.164Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 675b17aa378af168bbfd66ff
paper_authors.name: Shaohan Huang
paper_authors._id: 675b17aa378af168bbfd6700
paper_authors.name: Jianyong Wang
paper_authors._id: 675b17aa378af168bbfd6701
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:39:56.365Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Multimodal Latent Language Modeling with Next-Token Diffusion
paper_summary: Multimodal generative models require a unified approach to handle both
  discrete data (e.g., text and code) and continuous data (e.g., image, audio,
  video). In this work, we propose Latent Language Modeling (LatentLM), which
  seamlessly integrates continuous and discrete data using causal Transformers.
  Specifically, we employ a variational autoencoder (VAE) to represent continuous
  data as latent vectors and introduce next-token diffusion for autoregressive
  generation of these vectors. Additionally, we develop sigma-VAE to address
  the challenges of variance collapse, which is crucial for autoregressive
  modeling. Extensive experiments demonstrate the effectiveness of LatentLM
  across various modalities. In image generation, LatentLM surpasses Diffusion
  Transformers in both performance and scalability. When integrated into
  multimodal large language models, LatentLM provides a general-purpose interface
  that unifies multimodal generation and understanding. Experimental results show
  that LatentLM achieves favorable performance compared to Transfusion and vector
  quantized models in the setting of scaling up training tokens. In
  text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2
  model in speaker similarity and robustness, while requiring 10x fewer decoding
  steps. The results establish LatentLM as a highly effective and scalable
  approach to advance large multimodal models.
paper_ai_keywords: Latent Language Modeling (LatentLM) causal Transformers variational autoencoder (VAE) latent vectors next-token diffusion $\sigma$-VAE variance collapse Diffusion Transformers Transfusion vector quantized models text-to-speech synthesis VALL-E 2 model speaker similarity robustness