title: Next Token Prediction Towards Multimodal Intelligence: A Comprehensive
    Survey
summary: Building on the foundations of language modeling in natural language
  processing, Next Token Prediction (NTP) has evolved into a versatile training
  objective for machine learning tasks across various modalities, achieving
  considerable success. As Large Language Models (LLMs) have advanced to unify
  understanding and generation tasks within the textual modality, recent research
  has shown that tasks from different modalities can also be effectively
  encapsulated within the NTP framework, transforming the multimodal information
  into tokens and predict the next one given the context. This survey introduces
  a comprehensive taxonomy that unifies both understanding and generation within
  multimodal learning through the lens of NTP. The proposed taxonomy covers five
  key aspects: Multimodal tokenization, MMNTP model architectures, unified task
  representation, datasets \& evaluation, and open challenges. This new taxonomy
  aims to aid researchers in their exploration of multimodal intelligence. An
  associated GitHub repository collecting the latest papers and repos is
  available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction
paper_authors._id: 67720aa292c63806bde6d2be
paper_authors.name: Liang Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-30T02:51:16.924Z
paper_authors._id: 61b0a4ce1b3d95b3d1ed9251
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: leonardPKU
paper_authors._id: 67720aa292c63806bde6d2bf
paper_authors.name: Zekun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:26:13.884Z
paper_authors._id: 656832dfbd65fd41ee7aa8cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
paper_authors.fullname: Zekun Wang
paper_authors.type: user
paper_authors.user: kugwzk
paper_authors._id: 67720aa292c63806bde6d2c0
paper_authors.name: Shuhuai Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:26:21.168Z
paper_authors._id: 60d2e681b8448e1785bbda06
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624434302056-noauth.jpeg
paper_authors.fullname: Shuhuai Ren
paper_authors.type: user
paper_authors.user: ShuhuaiRen
paper_authors._id: 67720aa292c63806bde6d2c1
paper_authors.name: Lei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:13:41.398Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 67720aa292c63806bde6d2c2
paper_authors.name: Haozhe Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:43:49.539Z
paper_authors._id: 63ff09a02098b9ad105a09f6
paper_authors.avatarUrl: /avatars/4409ca5d320050cf4c3df05962c7ff58.svg
paper_authors.fullname: Hans Zhao
paper_authors.type: user
paper_authors.user: BleachNick
paper_authors._id: 67720aa292c63806bde6d2c3
paper_authors.name: Yunshui Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:30:44.113Z
paper_authors._id: 62e670d33651180f7d334ef3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
paper_authors.fullname: Li Yunshui
paper_authors.type: user
paper_authors.user: Wa2erGo
paper_authors._id: 67720aa292c63806bde6d2c4
paper_authors.name: Zefan Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-02T10:19:21.147Z
paper_authors._id: 66f2f84bdc54099111af9c59
paper_authors.avatarUrl: /avatars/91c998f82f2ee99cff9cfff859f9c76e.svg
paper_authors.fullname: Zefan Cai
paper_authors.type: user
paper_authors.user: Zefan-Cai
paper_authors._id: 67720aa292c63806bde6d2c5
paper_authors.name: Hongcheng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:26:44.644Z
paper_authors._id: 60ff653699f651283c017df7
paper_authors.avatarUrl: /avatars/075257e8d7fefc43be24738f978ea5c3.svg
paper_authors.fullname: Hongcheng Guo
paper_authors.type: user
paper_authors.user: hongcheng
paper_authors._id: 67720aa292c63806bde6d2c6
paper_authors.name: Lei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:13:45.095Z
paper_authors._id: 64c38871f9cd765462fa1a17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
paper_authors.fullname: Lei Zhang
paper_authors.type: user
paper_authors.user: Lemoncoke
paper_authors._id: 67720aa292c63806bde6d2c7
paper_authors.name: Yizhe Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:34.303Z
paper_authors._id: 642e916f6a378e41aa561516
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Fsas7e23J8Fhexvi8LoSN.jpeg
paper_authors.fullname: Yizhe Xiong
paper_authors.type: user
paper_authors.user: Bostoncake
paper_authors._id: 67720aa292c63806bde6d2c8
paper_authors.name: Yichi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:24:36.386Z
paper_authors._id: 6369b1d456d1f93498130a8a
paper_authors.avatarUrl: /avatars/8ec228aa6f171715652511f948765db9.svg
paper_authors.fullname: Yichi Zhang
paper_authors.type: user
paper_authors.user: 594zyc
paper_authors._id: 67720aa292c63806bde6d2c9
paper_authors.name: Ruoyu Wu
paper_authors._id: 67720aa292c63806bde6d2ca
paper_authors.name: Qingxiu Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:25:52.373Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 67720aa292c63806bde6d2cb
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:26:59.568Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 67720aa292c63806bde6d2cc
paper_authors.name: Jian Yang
paper_authors._id: 67720aa292c63806bde6d2cd
paper_authors.name: Lingwei Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:25:39.412Z
paper_authors._id: 65d8d2d0182779056299f668
paper_authors.avatarUrl: /avatars/2f21723a6348ee55431336b740aac9d7.svg
paper_authors.fullname: Lingwei Meng
paper_authors.type: user
paper_authors.user: LingweiMeng
paper_authors._id: 67720aa292c63806bde6d2ce
paper_authors.name: Shujie Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:25:46.895Z
paper_authors._id: 62a9ed2e097098dc397b5b82
paper_authors.avatarUrl: /avatars/cb6b05ec338e4f46187151527157781d.svg
paper_authors.fullname: Shujie Hu
paper_authors.type: user
paper_authors.user: sjhu
paper_authors._id: 67720aa292c63806bde6d2cf
paper_authors.name: Yulong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:27:19.601Z
paper_authors._id: 627e07f953494d140baea7fd
paper_authors.avatarUrl: /avatars/e6c4cc361f0cbcd5e27840d477cd06f3.svg
paper_authors.fullname: cyl
paper_authors.type: user
paper_authors.user: yulongchen
paper_authors._id: 67720aa292c63806bde6d2d0
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:22:56.705Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 67720aa292c63806bde6d2d1
paper_authors.name: Shuai Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:27:34.048Z
paper_authors._id: 63451cf0a05b51f7ded25505
paper_authors.avatarUrl: /avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
paper_authors.fullname: shuai bai
paper_authors.type: user
paper_authors.user: bluelike
paper_authors._id: 67720aa292c63806bde6d2d2
paper_authors.name: Andreas Vlachos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T19:36:05.328Z
paper_authors._id: 66a638a54166878166e4169d
paper_authors.avatarUrl: /avatars/5b7c5b1654a0a6e6bc746ef2c54f2ab0.svg
paper_authors.fullname: Andreas Vlachos
paper_authors.type: user
paper_authors.user: andreasvlachos
paper_authors._id: 67720aa292c63806bde6d2d3
paper_authors.name: Xu Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:27:49.723Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 67720aa292c63806bde6d2d4
paper_authors.name: Minjia Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T09:28:01.219Z
paper_authors._id: 6305b5e39d2531fabd195c5f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6305b5e39d2531fabd195c5f/oGS5leXVrB8Ct_BF96VkR.jpeg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Minjia
paper_authors._id: 67720aa292c63806bde6d2d5
paper_authors.name: Wen Xiao
paper_authors._id: 67720aa292c63806bde6d2d6
paper_authors.name: Aaron Yee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T09:47:35.995Z
paper_authors._id: 65f970a362c5227d533fe56b
paper_authors.avatarUrl: /avatars/c54eb2b41bfe9005cfb6b98b5451d784.svg
paper_authors.fullname: Aaron Yee
paper_authors.type: user
paper_authors.user: aaronyee
paper_authors._id: 67720aa292c63806bde6d2d7
paper_authors.name: Tianyu Liu
paper_authors._id: 67720aa292c63806bde6d2d8
paper_authors.name: Baobao Chang
paper_title: Next Token Prediction Towards Multimodal Intelligence: A Comprehensive
    Survey
paper_summary: Building on the foundations of language modeling in natural language
  processing, Next Token Prediction (NTP) has evolved into a versatile training
  objective for machine learning tasks across various modalities, achieving
  considerable success. As Large Language Models (LLMs) have advanced to unify
  understanding and generation tasks within the textual modality, recent research
  has shown that tasks from different modalities can also be effectively
  encapsulated within the NTP framework, transforming the multimodal information
  into tokens and predict the next one given the context. This survey introduces
  a comprehensive taxonomy that unifies both understanding and generation within
  multimodal learning through the lens of NTP. The proposed taxonomy covers five
  key aspects: Multimodal tokenization, MMNTP model architectures, unified task
  representation, datasets \& evaluation, and open challenges. This new taxonomy
  aims to aid researchers in their exploration of multimodal intelligence. An
  associated GitHub repository collecting the latest papers and repos is
  available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction
paper_ai_summary: A comprehensive taxonomy unifies multimodal understanding and generation through Next Token Prediction, covering aspects like tokenization, model architectures, task representation, datasets, and open challenges.
paper_ai_keywords: Next Token Prediction NTP Large Language Models LLMs multimodal learning multimodal tokenization MMNTP unified task representation datasets evaluation