Next Token Prediction Towards Multimodal Intelligence: A Comprehensive
  Survey
Building on the foundations of language modeling in natural language
processing, Next Token Prediction (NTP) has evolved into a versatile training
objective for machine learning tasks across various modalities, achieving
considerable success. As Large Language Models (LLMs) have advanced to unify
understanding and generation tasks within the textual modality, recent research
has shown that tasks from different modalities can also be effectively
encapsulated within the NTP framework, transforming the multimodal information
into tokens and predict the next one given the context. This survey introduces
a comprehensive taxonomy that unifies both understanding and generation within
multimodal learning through the lens of NTP. The proposed taxonomy covers five
key aspects: Multimodal tokenization, MMNTP model architectures, unified task
representation, datasets \& evaluation, and open challenges. This new taxonomy
aims to aid researchers in their exploration of multimodal intelligence. An
associated GitHub repository collecting the latest papers and repos is
available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction
67720aa292c63806bde6d2be
Liang Chen
extracted_pending
2024-12-30T02:51:16.924Z
61b0a4ce1b3d95b3d1ed9251
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
Liang Chen
user
leonardPKU
67720aa292c63806bde6d2bf
Zekun Wang
admin_assigned
2025-01-10T09:26:13.884Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
67720aa292c63806bde6d2c0
Shuhuai Ren
admin_assigned
2025-01-10T09:26:21.168Z
60d2e681b8448e1785bbda06
https://cdn-avatars.huggingface.co/v1/production/uploads/1624434302056-noauth.jpeg
Shuhuai Ren
user
ShuhuaiRen
67720aa292c63806bde6d2c1
Lei Li
claimed_verified
2025-05-13T08:13:41.398Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
67720aa292c63806bde6d2c2
Haozhe Zhao
claimed_verified
2025-05-26T08:43:49.539Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
67720aa292c63806bde6d2c3
Yunshui Li
claimed_verified
2024-12-30T19:30:44.113Z
62e670d33651180f7d334ef3
https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
Li Yunshui
user
Wa2erGo
67720aa292c63806bde6d2c4
Zefan Cai
claimed_verified
2025-01-02T10:19:21.147Z
66f2f84bdc54099111af9c59
/avatars/91c998f82f2ee99cff9cfff859f9c76e.svg
Zefan Cai
user
Zefan-Cai
67720aa292c63806bde6d2c5
Hongcheng Guo
admin_assigned
2025-01-10T09:26:44.644Z
60ff653699f651283c017df7
/avatars/075257e8d7fefc43be24738f978ea5c3.svg
Hongcheng Guo
user
hongcheng
67720aa292c63806bde6d2c6
Lei Zhang
claimed_verified
2025-01-01T20:13:45.095Z
64c38871f9cd765462fa1a17
https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
Lei Zhang
user
Lemoncoke
67720aa292c63806bde6d2c7
Yizhe Xiong
claimed_verified
2025-01-03T14:05:34.303Z
642e916f6a378e41aa561516
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Fsas7e23J8Fhexvi8LoSN.jpeg
Yizhe Xiong
user
Bostoncake
67720aa292c63806bde6d2c8
Yichi Zhang
admin_assigned
2025-01-10T09:24:36.386Z
6369b1d456d1f93498130a8a
/avatars/8ec228aa6f171715652511f948765db9.svg
Yichi Zhang
user
594zyc
67720aa292c63806bde6d2c9
Ruoyu Wu
67720aa292c63806bde6d2ca
Qingxiu Dong
admin_assigned
2025-01-10T09:25:52.373Z
670740744341dcee459fb990
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
Qingxiu Dong
user
Rsy24
67720aa292c63806bde6d2cb
Ge Zhang
admin_assigned
2025-01-10T09:26:59.568Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
67720aa292c63806bde6d2cc
Jian Yang
67720aa292c63806bde6d2cd
Lingwei Meng
admin_assigned
2025-01-10T09:25:39.412Z
65d8d2d0182779056299f668
/avatars/2f21723a6348ee55431336b740aac9d7.svg
Lingwei Meng
user
LingweiMeng
67720aa292c63806bde6d2ce
Shujie Hu
admin_assigned
2025-01-10T09:25:46.895Z
62a9ed2e097098dc397b5b82
/avatars/cb6b05ec338e4f46187151527157781d.svg
Shujie Hu
user
sjhu
67720aa292c63806bde6d2cf
Yulong Chen
admin_assigned
2025-01-10T09:27:19.601Z
627e07f953494d140baea7fd
/avatars/e6c4cc361f0cbcd5e27840d477cd06f3.svg
cyl
user
yulongchen
67720aa292c63806bde6d2d0
Junyang Lin
admin_assigned
2025-01-10T09:22:56.705Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
67720aa292c63806bde6d2d1
Shuai Bai
admin_assigned
2025-01-10T09:27:34.048Z
63451cf0a05b51f7ded25505
/avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
shuai bai
user
bluelike
67720aa292c63806bde6d2d2
Andreas Vlachos
admin_assigned
2024-12-30T19:36:05.328Z
66a638a54166878166e4169d
/avatars/5b7c5b1654a0a6e6bc746ef2c54f2ab0.svg
Andreas Vlachos
user
andreasvlachos
67720aa292c63806bde6d2d3
Xu Tan
admin_assigned
2025-01-10T09:27:49.723Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
67720aa292c63806bde6d2d4
Minjia Zhang
admin_assigned
2025-01-10T09:28:01.219Z
6305b5e39d2531fabd195c5f
https://cdn-avatars.huggingface.co/v1/production/uploads/6305b5e39d2531fabd195c5f/oGS5leXVrB8Ct_BF96VkR.jpeg
Zhang
user
Minjia
67720aa292c63806bde6d2d5
Wen Xiao
67720aa292c63806bde6d2d6
Aaron Yee
claimed_verified
2025-01-08T09:47:35.995Z
65f970a362c5227d533fe56b
/avatars/c54eb2b41bfe9005cfb6b98b5451d784.svg
Aaron Yee
user
aaronyee
67720aa292c63806bde6d2d7
Tianyu Liu
67720aa292c63806bde6d2d8
Baobao Chang
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive
  Survey
Building on the foundations of language modeling in natural language
processing, Next Token Prediction (NTP) has evolved into a versatile training
objective for machine learning tasks across various modalities, achieving
considerable success. As Large Language Models (LLMs) have advanced to unify
understanding and generation tasks within the textual modality, recent research
has shown that tasks from different modalities can also be effectively
encapsulated within the NTP framework, transforming the multimodal information
into tokens and predict the next one given the context. This survey introduces
a comprehensive taxonomy that unifies both understanding and generation within
multimodal learning through the lens of NTP. The proposed taxonomy covers five
key aspects: Multimodal tokenization, MMNTP model architectures, unified task
representation, datasets \& evaluation, and open challenges. This new taxonomy
aims to aid researchers in their exploration of multimodal intelligence. An
associated GitHub repository collecting the latest papers and repos is
available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction
A comprehensive taxonomy unifies multimodal understanding and generation through Next Token Prediction, covering aspects like tokenization, model architectures, task representation, datasets, and open challenges.
Next Token Prediction NTP Large Language Models LLMs multimodal learning multimodal tokenization MMNTP unified task representation datasets evaluation