title: Omni-RGPT: Unifying Image and Video Region-level Understanding via Token
    Marks
summary: We present Omni-RGPT, a multimodal large language model designed to
  facilitate region-level comprehension for both images and videos. To achieve
  consistent region representation across spatio-temporal dimensions, we
  introduce Token Mark, a set of tokens highlighting the target regions within
  the visual feature space. These tokens are directly embedded into spatial
  regions using region prompts (e.g., boxes or masks) and simultaneously
  incorporated into the text prompt to specify the target, establishing a direct
  connection between visual and text tokens. To further support robust video
  understanding without requiring tracklets, we introduce an auxiliary task that
  guides Token Mark by leveraging the consistency of the tokens, enabling stable
  region interpretation across the video. Additionally, we introduce a
  large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT
  achieves state-of-the-art results on image and video-based commonsense
  reasoning benchmarks while showing strong performance in captioning and
  referring expression comprehension tasks.
paper_authors._id: 6787772516c02260b4e6592b
paper_authors.name: Miran Heo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:55:26.369Z
paper_authors._id: 64b0640d99fb025c573d1bfc
paper_authors.avatarUrl: /avatars/b022f9854af8cd165c6282174dc8bdc5.svg
paper_authors.fullname: Miran Heo
paper_authors.type: user
paper_authors.user: miranheo
paper_authors._id: 6787772516c02260b4e6592c
paper_authors.name: Min-Hung Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T16:35:59.983Z
paper_authors._id: 64ae22dd1aee69ece065cdcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
paper_authors.fullname: Min-Hung Chen
paper_authors.type: user
paper_authors.user: cmhungsteve
paper_authors._id: 6787772516c02260b4e6592d
paper_authors.name: De-An Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:55:31.462Z
paper_authors._id: 641d1c5ec3983aa94915c162
paper_authors.avatarUrl: /avatars/127985b837ecf61e43c835deee578b5e.svg
paper_authors.fullname: De-An Huang
paper_authors.type: user
paper_authors.user: deahuang
paper_authors._id: 6787772516c02260b4e6592e
paper_authors.name: Sifei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:55:45.921Z
paper_authors._id: 62fab69f8cd542e895bafd6e
paper_authors.avatarUrl: /avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
paper_authors.fullname: Sifei Liu
paper_authors.type: user
paper_authors.user: zwrq
paper_authors._id: 6787772516c02260b4e6592f
paper_authors.name: Subhashree Radhakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:55:54.632Z
paper_authors._id: 63b609f50d5913eee489be7a
paper_authors.avatarUrl: /avatars/2d1a6cdd11a76d5c03a97bbf8c52b7e5.svg
paper_authors.fullname: Subhashree Radhakrishnan
paper_authors.type: user
paper_authors.user: subhashreer
paper_authors._id: 6787772516c02260b4e65930
paper_authors.name: Seon Joo Kim
paper_authors._id: 6787772516c02260b4e65931
paper_authors.name: Yu-Chiang Frank Wang
paper_authors._id: 6787772516c02260b4e65932
paper_authors.name: Ryo Hachiuma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:56:10.401Z
paper_authors._id: 65b33e5f7cd0069ad648c4e8
paper_authors.avatarUrl: /avatars/1a746ea535cffa92ea08006e05ea414a.svg
paper_authors.fullname: Ryo Hachiuma
paper_authors.type: user
paper_authors.user: rhachiuma
paper_title: Omni-RGPT: Unifying Image and Video Region-level Understanding via Token
    Marks
paper_summary: We present Omni-RGPT, a multimodal large language model designed to
  facilitate region-level comprehension for both images and videos. To achieve
  consistent region representation across spatio-temporal dimensions, we
  introduce Token Mark, a set of tokens highlighting the target regions within
  the visual feature space. These tokens are directly embedded into spatial
  regions using region prompts (e.g., boxes or masks) and simultaneously
  incorporated into the text prompt to specify the target, establishing a direct
  connection between visual and text tokens. To further support robust video
  understanding without requiring tracklets, we introduce an auxiliary task that
  guides Token Mark by leveraging the consistency of the tokens, enabling stable
  region interpretation across the video. Additionally, we introduce a
  large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT
  achieves state-of-the-art results on image and video-based commonsense
  reasoning benchmarks while showing strong performance in captioning and
  referring expression comprehension tasks.
paper_ai_keywords: multimodal large language model region-level comprehension Token Mark visual feature space region prompts text prompt video understanding auxiliary task consistency region interpretation large-scale region-level video instruction dataset RegVID-300k image-based commonsense reasoning video-based commonsense reasoning captioning referring expression comprehension