title: Emu3: Next-Token Prediction is All You Need
summary: While next-token prediction is considered a promising path towards artificial
  general intelligence, it has struggled to excel in multimodal tasks, which are
  still dominated by diffusion models (e.g., Stable Diffusion) and compositional
  approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a
  new suite of state-of-the-art multimodal models trained solely with next-token
  prediction. By tokenizing images, text, and videos into a discrete space, we
  train a single transformer from scratch on a mixture of multimodal sequences.
  Emu3 outperforms several well-established task-specific models in both
  generation and perception tasks, surpassing flagship models such as SDXL and
  LLaVA-1.6, while eliminating the need for diffusion or compositional
  architectures. Emu3 is also capable of generating high-fidelity video via
  predicting the next token in a video sequence. We simplify complex multimodal
  model designs by converging on a singular focus: tokens, unlocking great
  potential for scaling both during training and inference. Our results
  demonstrate that next-token prediction is a promising path towards building
  general multimodal intelligence beyond language. We open-source key techniques
  and models to support further research in this direction.
paper_authors._id: 66fa2b34290c1dc129add7a8
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:14:46.256Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_authors._id: 66fa2b34290c1dc129add7a9
paper_authors.name: Xiaosong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:15:11.793Z
paper_authors._id: 64e473f0b78bc92221ab1883
paper_authors.avatarUrl: /avatars/85dd02c2b4879d7696b552f0f9dc2680.svg
paper_authors.fullname: Xiaosong Zhang
paper_authors.type: user
paper_authors.user: zhangxiaosong18
paper_authors._id: 66fa2b34290c1dc129add7aa
paper_authors.name: Zhengxiong Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T08:34:37.930Z
paper_authors._id: 64afbb93bbdbfc1b8921960b
paper_authors.avatarUrl: /avatars/9ed45fa8a051c10cddfe2c9a3624d619.svg
paper_authors.fullname: zhengxiong
paper_authors.type: user
paper_authors.user: lzxlog
paper_authors._id: 66fa2b34290c1dc129add7ab
paper_authors.name: Quan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:15:36.626Z
paper_authors._id: 630d7a8f81ef9b1772b67f4c
paper_authors.avatarUrl: /avatars/00757abd6e548ccebb5bfb233be129a2.svg
paper_authors.fullname: Quan Sun
paper_authors.type: user
paper_authors.user: QuanSun
paper_authors._id: 66fa2b34290c1dc129add7ac
paper_authors.name: Yufeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:15:42.953Z
paper_authors._id: 648683de623b5f050213f2be
paper_authors.avatarUrl: /avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
paper_authors.fullname: Yufeng Cui
paper_authors.type: user
paper_authors.user: YufengCui
paper_authors._id: 66fa2b34290c1dc129add7ad
paper_authors.name: Jinsheng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:06:05.320Z
paper_authors._id: 6444a987af034cdfd69f6f8d
paper_authors.avatarUrl: /avatars/36544a102771a80c4fc6cb39e0973b2f.svg
paper_authors.fullname: Jinsheng Wang
paper_authors.type: user
paper_authors.user: wolfwjs
paper_authors._id: 66fa2b34290c1dc129add7ae
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:21:23.975Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 66fa2b34290c1dc129add7af
paper_authors.name: Yueze Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:16:08.897Z
paper_authors._id: 6458b59c7a7e192202df8fa0
paper_authors.avatarUrl: /avatars/33ee716477e5686da8723d01e199cd27.svg
paper_authors.fullname: Yueze Wang
paper_authors.type: user
paper_authors.user: yzwang
paper_authors._id: 66fa2b34290c1dc129add7b0
paper_authors.name: Zhen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:06:07.133Z
paper_authors._id: 6285a9133ab6642179158944
paper_authors.avatarUrl: /avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
paper_authors.fullname: Zhen Li
paper_authors.type: user
paper_authors.user: Paper99
paper_authors._id: 66fa2b34290c1dc129add7b1
paper_authors.name: Qiying Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:16:14.863Z
paper_authors._id: 6341375be5071b0c5cc946a3
paper_authors.avatarUrl: /avatars/c26a32b54a3f7731854059d247b5c523.svg
paper_authors.fullname: Qiying Yu
paper_authors.type: user
paper_authors.user: qiying
paper_authors._id: 66fa2b34290c1dc129add7b2
paper_authors.name: Yingli Zhao
paper_authors._id: 66fa2b34290c1dc129add7b3
paper_authors.name: Yulong Ao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:16:31.716Z
paper_authors._id: 64f3eb4fd41e83d74a0173ec
paper_authors.avatarUrl: /avatars/461f804bb5a7c5b31f862cfa27e52de1.svg
paper_authors.fullname: Yulong Ao
paper_authors.type: user
paper_authors.user: aoyulong
paper_authors._id: 66fa2b34290c1dc129add7b4
paper_authors.name: Xuebin Min
paper_authors._id: 66fa2b34290c1dc129add7b5
paper_authors.name: Tao Li
paper_authors._id: 66fa2b34290c1dc129add7b6
paper_authors.name: Boya Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:17:07.155Z
paper_authors._id: 64acaeccb575c5e272f5f124
paper_authors.avatarUrl: /avatars/29b0f9d0ad38755a52db1e69d112caa9.svg
paper_authors.fullname: BoyaWu
paper_authors.type: user
paper_authors.user: BoyaWu10
paper_authors._id: 66fa2b34290c1dc129add7b7
paper_authors.name: Bo Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:17:27.547Z
paper_authors._id: 64acafac1aee69ece03af05a
paper_authors.avatarUrl: /avatars/6380777852546dbd432fed5544d5db0b.svg
paper_authors.fullname: Bo Zhao
paper_authors.type: user
paper_authors.user: Z-LIRA
paper_authors._id: 66fa2b34290c1dc129add7b8
paper_authors.name: Bowen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:30.817Z
paper_authors._id: 6335113375bed9932474315e
paper_authors.avatarUrl: /avatars/aa1837a3a07514b89c48de888934a7b2.svg
paper_authors.fullname: bowenzhang
paper_authors.type: user
paper_authors.user: bowen92
paper_authors._id: 66fa2b34290c1dc129add7b9
paper_authors.name: Liangdong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:16:00.595Z
paper_authors._id: 63a11ce02fabbbb899a01d58
paper_authors.avatarUrl: /avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
paper_authors.fullname: ldwang
paper_authors.type: user
paper_authors.user: ldwang
paper_authors._id: 66fa2b34290c1dc129add7ba
paper_authors.name: Guang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:27:03.763Z
paper_authors._id: 632c234f42c386ebd2710434
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
paper_authors.fullname: Guang Liu
paper_authors.type: user
paper_authors.user: ZacLiu
paper_authors._id: 66fa2b34290c1dc129add7bb
paper_authors.name: Zheqi He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:18:32.699Z
paper_authors._id: 65b21047f5d76208991e463e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b21047f5d76208991e463e/6ML4lLz-vUr1HdWR3Jo-L.jpeg
paper_authors.fullname: Zheqi He
paper_authors.type: user
paper_authors.user: philokey
paper_authors._id: 66fa2b34290c1dc129add7bc
paper_authors.name: Xi Yang
paper_authors._id: 66fa2b34290c1dc129add7bd
paper_authors.name: Jingjing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:19:17.332Z
paper_authors._id: 650bf18678d2dfbad85c0db1
paper_authors.avatarUrl: /avatars/07ac9d726dc2ccf433746ee858a8044f.svg
paper_authors.fullname: jin liu
paper_authors.type: user
paper_authors.user: jingjingliu
paper_authors._id: 66fa2b34290c1dc129add7be
paper_authors.name: Yonghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T08:19:23.808Z
paper_authors._id: 629aa3155ab4232a3fe0893e
paper_authors.avatarUrl: /avatars/cf2d4a9295b5da9e2e4d2278bbb36040.svg
paper_authors.fullname: Yonghua Lin
paper_authors.type: user
paper_authors.user: Yonghua
paper_authors._id: 66fa2b34290c1dc129add7bf
paper_authors.name: Tiejun Huang
paper_authors._id: 66fa2b34290c1dc129add7c0
paper_authors.name: Zhongyuan Wang
paper_title: Emu3: Next-Token Prediction is All You Need
paper_summary: While next-token prediction is considered a promising path towards artificial
  general intelligence, it has struggled to excel in multimodal tasks, which are
  still dominated by diffusion models (e.g., Stable Diffusion) and compositional
  approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a
  new suite of state-of-the-art multimodal models trained solely with next-token
  prediction. By tokenizing images, text, and videos into a discrete space, we
  train a single transformer from scratch on a mixture of multimodal sequences.
  Emu3 outperforms several well-established task-specific models in both
  generation and perception tasks, surpassing flagship models such as SDXL and
  LLaVA-1.6, while eliminating the need for diffusion or compositional
  architectures. Emu3 is also capable of generating high-fidelity video via
  predicting the next token in a video sequence. We simplify complex multimodal
  model designs by converging on a singular focus: tokens, unlocking great
  potential for scaling both during training and inference. Our results
  demonstrate that next-token prediction is a promising path towards building
  general multimodal intelligence beyond language. We open-source key techniques
  and models to support further research in this direction.
paper_ai_summary: Emu3, a transformer-based multimodal model trained exclusively with next-token prediction, outperforms existing diffusion and compositional models in generation and perception tasks.
paper_ai_keywords: next-token prediction multimodal models transformer discrete space multimodal sequences high-fidelity video tokenization SDXL LLaVA-1.6