title: Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual
    Prompt Instruction Tuning
summary: Large Multimodal Models (LMMs) have made significant breakthroughs with the
  advancement of instruction tuning. However, while existing models can
  understand images and videos at a holistic level, they still struggle with
  instance-level understanding that requires a more nuanced comprehension and
  alignment. Instance-level understanding is crucial, as it focuses on the
  specific elements that we are most interested in. Excitingly, existing works
  find that the state-of-the-art LMMs exhibit strong instance understanding
  capabilities when provided with explicit visual cues. Motivated by this, we
  introduce an automated annotation pipeline assisted by GPT-4o to extract
  instance-level information from images and videos through explicit visual
  prompting for instance guidance. Building upon this pipeline, we proposed
  Inst-IT, a solution to enhance LMMs in Instance understanding via explicit
  visual prompt Instruction Tuning. Inst-IT consists of a benchmark to diagnose
  multimodal instance-level understanding, a large-scale instruction-tuning
  dataset, and a continuous instruction-tuning training paradigm to effectively
  enhance spatial-temporal instance understanding capabilities of existing LMMs.
  Experimental results show that, with the boost of Inst-IT, our models not only
  achieve outstanding performance on Inst-IT Bench but also demonstrate
  significant improvements across various generic image and video understanding
  benchmarks. This highlights that our dataset not only boosts instance-level
  understanding but also strengthens the overall capabilities of generic image
  and video comprehension.
paper_authors._id: 67519ac329945767b6c2ceeb
paper_authors.name: Wujian Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T12:22:26.413Z
paper_authors._id: 649bce4f200e2dff194d9883
paper_authors.avatarUrl: /avatars/b55a8bdc6f7e2bf9de5f26dc1d87bee3.svg
paper_authors.fullname: Wujian Peng
paper_authors.type: user
paper_authors.user: wjpoom
paper_authors._id: 67519ac329945767b6c2ceec
paper_authors.name: Lingchen Meng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:49.150Z
paper_authors._id: 64d852a4bab152b2470bf96e
paper_authors.avatarUrl: /avatars/074ef416ea865ca97c208ab2ba55d622.svg
paper_authors.fullname: Lingchen Meng
paper_authors.type: user
paper_authors.user: menglc
paper_authors._id: 67519ac329945767b6c2ceed
paper_authors.name: Yitong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:22:20.812Z
paper_authors._id: 64651db3611ae99d14d392ea
paper_authors.avatarUrl: /avatars/b818dc0dddc999758ab5737d5053e8c3.svg
paper_authors.fullname: cyt
paper_authors.type: user
paper_authors.user: Row11n
paper_authors._id: 67519ac329945767b6c2ceee
paper_authors.name: Yiweng Xie
paper_authors._id: 67519ac329945767b6c2ceef
paper_authors.name: Yang Liu
paper_authors._id: 67519ac329945767b6c2cef0
paper_authors.name: Tao Gui
paper_authors._id: 67519ac329945767b6c2cef1
paper_authors.name: Hang Xu
paper_authors._id: 67519ac329945767b6c2cef2
paper_authors.name: Xipeng Qiu
paper_authors._id: 67519ac329945767b6c2cef3
paper_authors.name: Zuxuan Wu
paper_authors._id: 67519ac329945767b6c2cef4
paper_authors.name: Yu-Gang Jiang
paper_title: Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual
    Prompt Instruction Tuning
paper_summary: Large Multimodal Models (LMMs) have made significant breakthroughs with the
  advancement of instruction tuning. However, while existing models can
  understand images and videos at a holistic level, they still struggle with
  instance-level understanding that requires a more nuanced comprehension and
  alignment. Instance-level understanding is crucial, as it focuses on the
  specific elements that we are most interested in. Excitingly, existing works
  find that the state-of-the-art LMMs exhibit strong instance understanding
  capabilities when provided with explicit visual cues. Motivated by this, we
  introduce an automated annotation pipeline assisted by GPT-4o to extract
  instance-level information from images and videos through explicit visual
  prompting for instance guidance. Building upon this pipeline, we proposed
  Inst-IT, a solution to enhance LMMs in Instance understanding via explicit
  visual prompt Instruction Tuning. Inst-IT consists of a benchmark to diagnose
  multimodal instance-level understanding, a large-scale instruction-tuning
  dataset, and a continuous instruction-tuning training paradigm to effectively
  enhance spatial-temporal instance understanding capabilities of existing LMMs.
  Experimental results show that, with the boost of Inst-IT, our models not only
  achieve outstanding performance on Inst-IT Bench but also demonstrate
  significant improvements across various generic image and video understanding
  benchmarks. This highlights that our dataset not only boosts instance-level
  understanding but also strengthens the overall capabilities of generic image
  and video comprehension.
paper_ai_summary: Inst-IT enhances large multimodal models' instance-level understanding through an automated annotation pipeline with explicit visual prompts, improving performance on specific elements in images and videos.
paper_ai_keywords: GPT-4o instance-level understanding automated annotation pipeline explicit visual prompting Inst-IT benchmark large-scale instruction-tuning dataset continuous instruction-tuning training paradigm spatial-temporal instance understanding