title: Kosmos-2.5: A Multimodal Literate Model
summary: We present Kosmos-2.5, a multimodal literate model for machine reading of
  text-intensive images. Pre-trained on large-scale text-intensive images,
  Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1)
  generating spatially-aware text blocks, where each block of text is assigned
  its spatial coordinates within the image, and (2) producing structured text
  output that captures styles and structures into the markdown format. This
  unified multimodal literate capability is achieved through a shared Transformer
  architecture, task-specific prompts, and flexible text representations. We
  evaluate Kosmos-2.5 on end-to-end document-level text recognition and
  image-to-markdown text generation. Furthermore, the model can be readily
  adapted for any text-intensive image understanding task with different prompts
  through supervised fine-tuning, making it a general-purpose tool for real-world
  applications involving text-rich images. This work also paves the way for the
  future scaling of multimodal large language models.
paper_authors._id: 650bc0752ec41a31f31b13e8
paper_authors.name: Tengchao Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:33:05.055Z
paper_authors._id: 614c7f86cbb5e52274a4024b
paper_authors.avatarUrl: /avatars/ff1a9079a1182d84397012fad746807c.svg
paper_authors.fullname: Tengchao Lv
paper_authors.type: user
paper_authors.user: estyle
paper_authors._id: 650bc0752ec41a31f31b13e9
paper_authors.name: Yupan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:41:39.856Z
paper_authors._id: 625d7fc87341c641426c7665
paper_authors.avatarUrl: /avatars/3083ebe0343a03332b15f0e7f76ba1d7.svg
paper_authors.fullname: Yupan Huang
paper_authors.type: user
paper_authors.user: HYPJUDY
paper_authors._id: 650bc0752ec41a31f31b13ea
paper_authors.name: Jingye Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:44:20.792Z
paper_authors._id: 63355049ce9a15ea6c4aa8cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63355049ce9a15ea6c4aa8cc/aD4NI5d_ozAuQVv5nsNF4.jpeg
paper_authors.fullname: JingyeChen
paper_authors.type: user
paper_authors.user: JingyeChen
paper_authors._id: 650bc0752ec41a31f31b13eb
paper_authors.name: Lei Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:44:50.418Z
paper_authors._id: 5f5a4fbe10b2753d9000c7fc
paper_authors.avatarUrl: /avatars/0763b25c13733ef4eb757862ba6d603e.svg
paper_authors.fullname: Lei Cui
paper_authors.type: user
paper_authors.user: wolfshow
paper_authors._id: 650bc0752ec41a31f31b13ec
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:44:56.951Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 650bc0752ec41a31f31b13ed
paper_authors.name: Yaoyao Chang
paper_authors._id: 650bc0752ec41a31f31b13ee
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:45:19.232Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 650bc0752ec41a31f31b13ef
paper_authors.name: Wenhui Wang
paper_authors._id: 650bc0752ec41a31f31b13f0
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:20:34.657Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 650bc0752ec41a31f31b13f1
paper_authors.name: Weiyao Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:46:17.438Z
paper_authors._id: 63a409a584a6a25c65bb3e2c
paper_authors.avatarUrl: /avatars/a08fb6052e7bde48cd3926c3d9bab30e.svg
paper_authors.fullname: Weiyao Luo
paper_authors.type: user
paper_authors.user: weiyao993
paper_authors._id: 650bc0752ec41a31f31b13f2
paper_authors.name: Shaoxiang Wu
paper_authors._id: 650bc0752ec41a31f31b13f3
paper_authors.name: Guoxin Wang
paper_authors._id: 650bc0752ec41a31f31b13f4
paper_authors.name: Cha Zhang
paper_authors._id: 650bc0752ec41a31f31b13f5
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:46:49.821Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Kosmos-2.5: A Multimodal Literate Model
paper_summary: We present Kosmos-2.5, a multimodal literate model for machine reading of
  text-intensive images. Pre-trained on large-scale text-intensive images,
  Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1)
  generating spatially-aware text blocks, where each block of text is assigned
  its spatial coordinates within the image, and (2) producing structured text
  output that captures styles and structures into the markdown format. This
  unified multimodal literate capability is achieved through a shared Transformer
  architecture, task-specific prompts, and flexible text representations. We
  evaluate Kosmos-2.5 on end-to-end document-level text recognition and
  image-to-markdown text generation. Furthermore, the model can be readily
  adapted for any text-intensive image understanding task with different prompts
  through supervised fine-tuning, making it a general-purpose tool for real-world
  applications involving text-rich images. This work also paves the way for the
  future scaling of multimodal large language models.
paper_ai_keywords: multimodal literate model machine reading pre-trained text-intensive images spatially-aware text blocks spatial coordinates structured text output markdown format shared Transformer architecture task-specific prompts flexible text representations end-to-end document-level text recognition image-to-markdown text generation supervised fine-tuning multimodal large language models