Emu3: Next-Token Prediction is All You Need
While next-token prediction is considered a promising path towards artificial
general intelligence, it has struggled to excel in multimodal tasks, which are
still dominated by diffusion models (e.g., Stable Diffusion) and compositional
approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a
new suite of state-of-the-art multimodal models trained solely with next-token
prediction. By tokenizing images, text, and videos into a discrete space, we
train a single transformer from scratch on a mixture of multimodal sequences.
Emu3 outperforms several well-established task-specific models in both
generation and perception tasks, surpassing flagship models such as SDXL and
LLaVA-1.6, while eliminating the need for diffusion or compositional
architectures. Emu3 is also capable of generating high-fidelity video via
predicting the next token in a video sequence. We simplify complex multimodal
model designs by converging on a singular focus: tokens, unlocking great
potential for scaling both during training and inference. Our results
demonstrate that next-token prediction is a promising path towards building
general multimodal intelligence beyond language. We open-source key techniques
and models to support further research in this direction.
66fa2b34290c1dc129add7a8
Xinlong Wang
admin_assigned
2024-09-30T08:14:46.256Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
66fa2b34290c1dc129add7a9
Xiaosong Zhang
admin_assigned
2024-09-30T08:15:11.793Z
64e473f0b78bc92221ab1883
/avatars/85dd02c2b4879d7696b552f0f9dc2680.svg
Xiaosong Zhang
user
zhangxiaosong18
66fa2b34290c1dc129add7aa
Zhengxiong Luo
claimed_verified
2024-11-25T08:34:37.930Z
64afbb93bbdbfc1b8921960b
/avatars/9ed45fa8a051c10cddfe2c9a3624d619.svg
zhengxiong
user
lzxlog
66fa2b34290c1dc129add7ab
Quan Sun
admin_assigned
2024-09-30T08:15:36.626Z
630d7a8f81ef9b1772b67f4c
/avatars/00757abd6e548ccebb5bfb233be129a2.svg
Quan Sun
user
QuanSun
66fa2b34290c1dc129add7ac
Yufeng Cui
admin_assigned
2024-09-30T08:15:42.953Z
648683de623b5f050213f2be
/avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
Yufeng Cui
user
YufengCui
66fa2b34290c1dc129add7ad
Jinsheng Wang
claimed_verified
2024-09-30T08:06:05.320Z
6444a987af034cdfd69f6f8d
/avatars/36544a102771a80c4fc6cb39e0973b2f.svg
Jinsheng Wang
user
wolfwjs
66fa2b34290c1dc129add7ae
Fan Zhang
claimed_verified
2024-10-08T07:21:23.975Z
640ed40dc025ddf618950af7
/avatars/a99abd4346fa649ad4144f284ebcc972.svg
Fan Zhang
user
ryanzhangfan
66fa2b34290c1dc129add7af
Yueze Wang
admin_assigned
2024-09-30T08:16:08.897Z
6458b59c7a7e192202df8fa0
/avatars/33ee716477e5686da8723d01e199cd27.svg
Yueze Wang
user
yzwang
66fa2b34290c1dc129add7b0
Zhen Li
claimed_verified
2024-09-30T08:06:07.133Z
6285a9133ab6642179158944
/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
Zhen Li
user
Paper99
66fa2b34290c1dc129add7b1
Qiying Yu
admin_assigned
2024-09-30T08:16:14.863Z
6341375be5071b0c5cc946a3
/avatars/c26a32b54a3f7731854059d247b5c523.svg
Qiying Yu
user
qiying
66fa2b34290c1dc129add7b2
Yingli Zhao
66fa2b34290c1dc129add7b3
Yulong Ao
admin_assigned
2024-09-30T08:16:31.716Z
64f3eb4fd41e83d74a0173ec
/avatars/461f804bb5a7c5b31f862cfa27e52de1.svg
Yulong Ao
user
aoyulong
66fa2b34290c1dc129add7b4
Xuebin Min
66fa2b34290c1dc129add7b5
Tao Li
66fa2b34290c1dc129add7b6
Boya Wu
admin_assigned
2024-09-30T08:17:07.155Z
64acaeccb575c5e272f5f124
/avatars/29b0f9d0ad38755a52db1e69d112caa9.svg
BoyaWu
user
BoyaWu10
66fa2b34290c1dc129add7b7
Bo Zhao
admin_assigned
2024-09-30T08:17:27.547Z
64acafac1aee69ece03af05a
/avatars/6380777852546dbd432fed5544d5db0b.svg
Bo Zhao
user
Z-LIRA
66fa2b34290c1dc129add7b8
Bowen Zhang
claimed_verified
2024-10-25T09:31:30.817Z
6335113375bed9932474315e
/avatars/aa1837a3a07514b89c48de888934a7b2.svg
bowenzhang
user
bowen92
66fa2b34290c1dc129add7b9
Liangdong Wang
claimed_verified
2024-10-24T07:16:00.595Z
63a11ce02fabbbb899a01d58
/avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
ldwang
user
ldwang
66fa2b34290c1dc129add7ba
Guang Liu
claimed_verified
2024-10-28T11:27:03.763Z
632c234f42c386ebd2710434
https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
Guang Liu
user
ZacLiu
66fa2b34290c1dc129add7bb
Zheqi He
admin_assigned
2024-09-30T08:18:32.699Z
65b21047f5d76208991e463e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b21047f5d76208991e463e/6ML4lLz-vUr1HdWR3Jo-L.jpeg
Zheqi He
user
philokey
66fa2b34290c1dc129add7bc
Xi Yang
66fa2b34290c1dc129add7bd
Jingjing Liu
admin_assigned
2024-09-30T08:19:17.332Z
650bf18678d2dfbad85c0db1
/avatars/07ac9d726dc2ccf433746ee858a8044f.svg
jin liu
user
jingjingliu
66fa2b34290c1dc129add7be
Yonghua Lin
admin_assigned
2024-09-30T08:19:23.808Z
629aa3155ab4232a3fe0893e
/avatars/cf2d4a9295b5da9e2e4d2278bbb36040.svg
Yonghua Lin
user
Yonghua
66fa2b34290c1dc129add7bf
Tiejun Huang
66fa2b34290c1dc129add7c0
Zhongyuan Wang
Emu3: Next-Token Prediction is All You Need
While next-token prediction is considered a promising path towards artificial
general intelligence, it has struggled to excel in multimodal tasks, which are
still dominated by diffusion models (e.g., Stable Diffusion) and compositional
approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a
new suite of state-of-the-art multimodal models trained solely with next-token
prediction. By tokenizing images, text, and videos into a discrete space, we
train a single transformer from scratch on a mixture of multimodal sequences.
Emu3 outperforms several well-established task-specific models in both
generation and perception tasks, surpassing flagship models such as SDXL and
LLaVA-1.6, while eliminating the need for diffusion or compositional
architectures. Emu3 is also capable of generating high-fidelity video via
predicting the next token in a video sequence. We simplify complex multimodal
model designs by converging on a singular focus: tokens, unlocking great
potential for scaling both during training and inference. Our results
demonstrate that next-token prediction is a promising path towards building
general multimodal intelligence beyond language. We open-source key techniques
and models to support further research in this direction.
Emu3, a transformer-based multimodal model trained exclusively with next-token prediction, outperforms existing diffusion and compositional models in generation and perception tasks.
next-token prediction multimodal models transformer discrete space multimodal sequences high-fidelity video tokenization SDXL LLaVA-1.6