title: Kosmos-2: Grounding Multimodal Large Language Models to the World
summary: We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new
  capabilities of perceiving object descriptions (e.g., bounding boxes) and
  grounding text to the visual world. Specifically, we represent refer
  expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where
  object descriptions are sequences of location tokens. Together with multimodal
  corpora, we construct large-scale data of grounded image-text pairs (called
  GrIT) to train the model. In addition to the existing capabilities of MLLMs
  (e.g., perceiving general modalities, following instructions, and performing
  in-context learning), Kosmos-2 integrates the grounding capability into
  downstream applications. We evaluate Kosmos-2 on a wide range of tasks,
  including (i) multimodal grounding, such as referring expression comprehension,
  and phrase grounding, (ii) multimodal referring, such as referring expression
  generation, (iii) perception-language tasks, and (iv) language understanding
  and generation. This work lays out the foundation for the development of
  Embodiment AI and sheds light on the big convergence of language, multimodal
  perception, action, and world modeling, which is a key step toward artificial
  general intelligence. Data, demo, and pretrained models are available at
  https://aka.ms/kosmos-2.
paper_authors._id: 649a54023c50d4ec5335193f
paper_authors.name: Zhiliang Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T07:24:20.810Z
paper_authors._id: 646c408336505117e22f4b36
paper_authors.avatarUrl: /avatars/8201da3c4ec51c8e113446f9578d44f6.svg
paper_authors.fullname: zhiliang
paper_authors.type: user
paper_authors.user: zzliang
paper_authors._id: 649a54023c50d4ec53351940
paper_authors.name: Wenhui Wang
paper_authors._id: 649a54023c50d4ec53351941
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T07:41:25.517Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 649a54023c50d4ec53351942
paper_authors.name: Yaru Hao
paper_authors._id: 649a54023c50d4ec53351943
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-27T13:08:34.909Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 649a54023c50d4ec53351944
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-27T13:08:06.367Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 649a54023c50d4ec53351945
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-27T13:07:46.648Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Kosmos-2: Grounding Multimodal Large Language Models to the World
paper_summary: We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new
  capabilities of perceiving object descriptions (e.g., bounding boxes) and
  grounding text to the visual world. Specifically, we represent refer
  expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where
  object descriptions are sequences of location tokens. Together with multimodal
  corpora, we construct large-scale data of grounded image-text pairs (called
  GrIT) to train the model. In addition to the existing capabilities of MLLMs
  (e.g., perceiving general modalities, following instructions, and performing
  in-context learning), Kosmos-2 integrates the grounding capability into
  downstream applications. We evaluate Kosmos-2 on a wide range of tasks,
  including (i) multimodal grounding, such as referring expression comprehension,
  and phrase grounding, (ii) multimodal referring, such as referring expression
  generation, (iii) perception-language tasks, and (iv) language understanding
  and generation. This work lays out the foundation for the development of
  Embodiment AI and sheds light on the big convergence of language, multimodal
  perception, action, and world modeling, which is a key step toward artificial
  general intelligence. Data, demo, and pretrained models are available at
  https://aka.ms/kosmos-2.
paper_ai_summary: Kosmos-2, a Multimodal Large Language Model, combines text-to-visual grounding with existing MLLM capabilities, enabling tasks like referring expression comprehension and phrase grounding.
paper_ai_keywords: Multimodal Large Language Model MLLM refer expressions Markdown location tokens grounded image-text pairs multimodal grounding referring expression comprehension phrase grounding multimodal referring referring expression generation perception-language tasks language understanding language generation Embodiment AI