Omni-RGPT: Unifying Image and Video Region-level Understanding via Token
  Marks
We present Omni-RGPT, a multimodal large language model designed to
facilitate region-level comprehension for both images and videos. To achieve
consistent region representation across spatio-temporal dimensions, we
introduce Token Mark, a set of tokens highlighting the target regions within
the visual feature space. These tokens are directly embedded into spatial
regions using region prompts (e.g., boxes or masks) and simultaneously
incorporated into the text prompt to specify the target, establishing a direct
connection between visual and text tokens. To further support robust video
understanding without requiring tracklets, we introduce an auxiliary task that
guides Token Mark by leveraging the consistency of the tokens, enabling stable
region interpretation across the video. Additionally, we introduce a
large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT
achieves state-of-the-art results on image and video-based commonsense
reasoning benchmarks while showing strong performance in captioning and
referring expression comprehension tasks.
6787772516c02260b4e6592b
Miran Heo
admin_assigned
2025-01-15T16:55:26.369Z
64b0640d99fb025c573d1bfc
/avatars/b022f9854af8cd165c6282174dc8bdc5.svg
Miran Heo
user
miranheo
6787772516c02260b4e6592c
Min-Hung Chen
claimed_verified
2025-01-15T16:35:59.983Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
6787772516c02260b4e6592d
De-An Huang
admin_assigned
2025-01-15T16:55:31.462Z
641d1c5ec3983aa94915c162
/avatars/127985b837ecf61e43c835deee578b5e.svg
De-An Huang
user
deahuang
6787772516c02260b4e6592e
Sifei Liu
admin_assigned
2025-01-15T16:55:45.921Z
62fab69f8cd542e895bafd6e
/avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
Sifei Liu
user
zwrq
6787772516c02260b4e6592f
Subhashree Radhakrishnan
admin_assigned
2025-01-15T16:55:54.632Z
63b609f50d5913eee489be7a
/avatars/2d1a6cdd11a76d5c03a97bbf8c52b7e5.svg
Subhashree Radhakrishnan
user
subhashreer
6787772516c02260b4e65930
Seon Joo Kim
6787772516c02260b4e65931
Yu-Chiang Frank Wang
6787772516c02260b4e65932
Ryo Hachiuma
admin_assigned
2025-01-15T16:56:10.401Z
65b33e5f7cd0069ad648c4e8
/avatars/1a746ea535cffa92ea08006e05ea414a.svg
Ryo Hachiuma
user
rhachiuma
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token
  Marks
We present Omni-RGPT, a multimodal large language model designed to
facilitate region-level comprehension for both images and videos. To achieve
consistent region representation across spatio-temporal dimensions, we
introduce Token Mark, a set of tokens highlighting the target regions within
the visual feature space. These tokens are directly embedded into spatial
regions using region prompts (e.g., boxes or masks) and simultaneously
incorporated into the text prompt to specify the target, establishing a direct
connection between visual and text tokens. To further support robust video
understanding without requiring tracklets, we introduce an auxiliary task that
guides Token Mark by leveraging the consistency of the tokens, enabling stable
region interpretation across the video. Additionally, we introduce a
large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT
achieves state-of-the-art results on image and video-based commonsense
reasoning benchmarks while showing strong performance in captioning and
referring expression comprehension tasks.
Omni-RGPT uses Token Mark with region prompts to achieve consistent and robust region-level multimodal understanding, supporting tasks like reasoning, captioning, and referring expression comprehension.
Token Mark region prompts region-level comprehension RegVID-300k commonsense reasoning captioning referring expression comprehension