title: MIO: A Foundation Model on Multimodal Tokens
summary: In this paper, we introduce MIO, a novel foundation model built on multimodal
  tokens, capable of understanding and generating speech, text, images, and
  videos in an end-to-end, autoregressive manner. While the emergence of large
  language models (LLMs) and multimodal large language models (MM-LLMs) propels
  advancements in artificial general intelligence through their versatile
  capabilities, they still lack true any-to-any understanding and generation.
  Recently, the release of GPT-4o has showcased the remarkable potential of
  any-to-any LLMs for complex real-world tasks, enabling omnidirectional input
  and output across images, speech, and text. However, it is closed-source and
  does not support the generation of multimodal interleaved sequences. To address
  this gap, we present MIO, which is trained on a mixture of discrete tokens
  across four modalities using causal multimodal modeling. MIO undergoes a
  four-stage training process: (1) alignment pre-training, (2) interleaved
  pre-training, (3) speech-enhanced pre-training, and (4) comprehensive
  supervised fine-tuning on diverse textual, visual, and speech tasks. Our
  experimental results indicate that MIO exhibits competitive, and in some cases
  superior, performance compared to previous dual-modal baselines, any-to-any
  model baselines, and even modality-specific baselines. Moreover, MIO
  demonstrates advanced capabilities inherent to its any-to-any feature, such as
  interleaved video-text generation, chain-of-visual-thought reasoning, visual
  guideline generation, instructional image editing, etc.
paper_authors._id: 66fa19f693b99e4c7c8a02ce
paper_authors.name: Zekun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:30:21.748Z
paper_authors._id: 6149a9e95347647e6bb68882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
paper_authors.fullname: Zekun Moore Wang
paper_authors.type: user
paper_authors.user: ZenMoore
paper_authors._id: 66fa19f693b99e4c7c8a02cf
paper_authors.name: King Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T09:25:03.209Z
paper_authors._id: 6578265ddea7e2122d02f6ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
paper_authors.fullname: kang zhu
paper_authors.type: user
paper_authors.user: kangz
paper_authors._id: 66fa19f693b99e4c7c8a02d0
paper_authors.name: Chunpu Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:26:31.334Z
paper_authors._id: 6580333b64bab467b9bcb615
paper_authors.avatarUrl: /avatars/753afae2fefc748791550c733aa2c80b.svg
paper_authors.fullname: Chunpu Xu
paper_authors.type: user
paper_authors.user: cpxu
paper_authors._id: 66fa19f693b99e4c7c8a02d1
paper_authors.name: Wangchunshu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:26:42.767Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 66fa19f693b99e4c7c8a02d2
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:26:48.968Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 66fa19f693b99e4c7c8a02d3
paper_authors.name: Yibo Zhang
paper_authors._id: 66fa19f693b99e4c7c8a02d4
paper_authors.name: Jiashuo Wang
paper_authors._id: 66fa19f693b99e4c7c8a02d5
paper_authors.name: Ning Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:28:12.312Z
paper_authors._id: 64e2f812bbee0c24b93fdb88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e2f812bbee0c24b93fdb88/UdeMJFDWjRjydJ38WjnnO.jpeg
paper_authors.fullname: Ning Shi
paper_authors.type: user
paper_authors.user: MrShininnnnn
paper_authors._id: 66fa19f693b99e4c7c8a02d6
paper_authors.name: Siyu Li
paper_authors._id: 66fa19f693b99e4c7c8a02d7
paper_authors.name: Yizhi Li
paper_authors._id: 66fa19f693b99e4c7c8a02d8
paper_authors.name: Haoran Que
paper_authors._id: 66fa19f693b99e4c7c8a02d9
paper_authors.name: Zhaoxiang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:29:05.696Z
paper_authors._id: 6350d59989def14ad21e11b3
paper_authors.avatarUrl: /avatars/9a7ae6131e0a89f3461f267ee844b218.svg
paper_authors.fullname: z
paper_authors.type: user
paper_authors.user: smallbert
paper_authors._id: 66fa19f693b99e4c7c8a02da
paper_authors.name: Yuanxing Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:29:51.984Z
paper_authors._id: 60ff5c28aa025227d344b734
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1627347980032-noauth.jpeg
paper_authors.fullname: Yuanxing Zhang
paper_authors.type: user
paper_authors.user: Onmyoji
paper_authors._id: 66fa19f693b99e4c7c8a02db
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:30:04.866Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66fa19f693b99e4c7c8a02dc
paper_authors.name: Ke Xu
paper_authors._id: 66fa19f693b99e4c7c8a02dd
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T07:16:54.588Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 66fa19f693b99e4c7c8a02de
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:30:10.239Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_title: MIO: A Foundation Model on Multimodal Tokens
paper_summary: In this paper, we introduce MIO, a novel foundation model built on multimodal
  tokens, capable of understanding and generating speech, text, images, and
  videos in an end-to-end, autoregressive manner. While the emergence of large
  language models (LLMs) and multimodal large language models (MM-LLMs) propels
  advancements in artificial general intelligence through their versatile
  capabilities, they still lack true any-to-any understanding and generation.
  Recently, the release of GPT-4o has showcased the remarkable potential of
  any-to-any LLMs for complex real-world tasks, enabling omnidirectional input
  and output across images, speech, and text. However, it is closed-source and
  does not support the generation of multimodal interleaved sequences. To address
  this gap, we present MIO, which is trained on a mixture of discrete tokens
  across four modalities using causal multimodal modeling. MIO undergoes a
  four-stage training process: (1) alignment pre-training, (2) interleaved
  pre-training, (3) speech-enhanced pre-training, and (4) comprehensive
  supervised fine-tuning on diverse textual, visual, and speech tasks. Our
  experimental results indicate that MIO exhibits competitive, and in some cases
  superior, performance compared to previous dual-modal baselines, any-to-any
  model baselines, and even modality-specific baselines. Moreover, MIO
  demonstrates advanced capabilities inherent to its any-to-any feature, such as
  interleaved video-text generation, chain-of-visual-thought reasoning, visual
  guideline generation, instructional image editing, etc.
paper_ai_keywords: multimodal tokens end-to-end autoregressive large language models (LLMs) multimodal large language models (MM-LLMs) artificial general intelligence any-to-any understanding any-to-any generation GPT-4o omnidirectional input omnidirectional output discrete tokens causal multimodal modeling alignment pre-training interleaved pre-training speech-enhanced pre-training comprehensive supervised fine-tuning dual-modal baselines any-to-any model baselines modality-specific baselines interleaved video-text generation chain-of-visual-thought reasoning visual guideline generation instructional image editing