VideoMolmo: Spatio-Temporal Grounding Meets Pointing
Spatio-temporal localization is vital for precise interactions across diverse
domains, from biological research to autonomous navigation and interactive
interfaces. Current video-based approaches, while proficient in tracking, lack
the sophisticated reasoning capabilities of large language models, limiting
their contextual understanding and generalization. We introduce VideoMolmo, a
large multimodal model tailored for fine-grained spatio-temporal pointing
conditioned on textual descriptions. Building upon the Molmo architecture,
VideoMolmo incorporates a temporal module utilizing an attention mechanism to
condition each frame on preceding frames, ensuring temporal consistency.
Additionally, our novel temporal mask fusion pipeline employs SAM2 for
bidirectional point propagation, significantly enhancing coherence across video
sequences. This two-step decomposition, i.e., first using the LLM to generate
precise pointing coordinates, then relying on a sequential mask-fusion module
to produce coherent segmentation, not only simplifies the task for the language
model but also enhances interpretability. Due to the lack of suitable datasets,
we curate a comprehensive dataset comprising 72k video-caption pairs annotated
with 100k object points. To evaluate the generalization of VideoMolmo, we
introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five
real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving,
Video-GUI Interaction, and Robotics. We also evaluate our model on Referring
Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to
existing models, VideoMolmo substantially improves spatio-temporal pointing
accuracy and reasoning capability. Our code and models are publicly available
at https://github.com/mbzuai-oryx/VideoMolmo.
68425a7ab63271ff41652734
Ghazi Shazan Ahmad
claimed_verified
2025-06-24T08:11:02.086Z
67121d145b7360457e3dcbda
/avatars/0d49e3231681571cc029a0a2b5b232a9.svg
GHAZI SHAZAN AHMAD
user
ghazishazan
68425a7ab63271ff41652735
Ahmed Heakl
claimed_verified
2025-06-07T05:48:53.401Z
656864e12d73834278a8dea7
https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg
Ahmed Heakl
user
ahmedheakl
68425a7ab63271ff41652736
Hanan Gani
claimed_verified
2025-06-24T08:11:04.406Z
64eddf17e630ae6d575f6231
https://cdn-avatars.huggingface.co/v1/production/uploads/64eddf17e630ae6d575f6231/W8oofVqStLEAK1Xi4fQTh.png
Hanan Gani
user
hanangani
68425a7ab63271ff41652737
Abdelrahman Shaker
68425a7ab63271ff41652738
Zhiqiang Shen
68425a7ab63271ff41652739
Ranjay Krishna
68425a7ab63271ff4165273a
Fahad Shahbaz Khan
68425a7ab63271ff4165273b
Salman Khan
VideoMolmo: Spatio-Temporal Grounding Meets Pointing
Spatio-temporal localization is vital for precise interactions across diverse
domains, from biological research to autonomous navigation and interactive
interfaces. Current video-based approaches, while proficient in tracking, lack
the sophisticated reasoning capabilities of large language models, limiting
their contextual understanding and generalization. We introduce VideoMolmo, a
large multimodal model tailored for fine-grained spatio-temporal pointing
conditioned on textual descriptions. Building upon the Molmo architecture,
VideoMolmo incorporates a temporal module utilizing an attention mechanism to
condition each frame on preceding frames, ensuring temporal consistency.
Additionally, our novel temporal mask fusion pipeline employs SAM2 for
bidirectional point propagation, significantly enhancing coherence across video
sequences. This two-step decomposition, i.e., first using the LLM to generate
precise pointing coordinates, then relying on a sequential mask-fusion module
to produce coherent segmentation, not only simplifies the task for the language
model but also enhances interpretability. Due to the lack of suitable datasets,
we curate a comprehensive dataset comprising 72k video-caption pairs annotated
with 100k object points. To evaluate the generalization of VideoMolmo, we
introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five
real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving,
Video-GUI Interaction, and Robotics. We also evaluate our model on Referring
Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to
existing models, VideoMolmo substantially improves spatio-temporal pointing
accuracy and reasoning capability. Our code and models are publicly available
at https://github.com/mbzuai-oryx/VideoMolmo.
VideoMolmo, a multimodal model incorporating a temporal attention mechanism and SAM2 for mask fusion, enhances spatio-temporal pointing accuracy and reasoning capabilities in diverse real-world scenarios.
Molmo attention mechanism temporal mask fusion SAM2 bidirectional point propagation VideoMolmo LLM sequential mask-fusion module VPoS-Bench Referring Video Object Segmentation Reasoning VOS