Caption Anything in Video: Fine-grained Object-centric Captioning via
  Spatiotemporal Multimodal Prompting
We present CAT-V (Caption AnyThing in Video), a training-free framework for
fine-grained object-centric video captioning that enables detailed descriptions
of user-selected objects through time. CAT-V integrates three key components: a
Segmenter based on SAMURAI for precise object segmentation across frames, a
Temporal Analyzer powered by TRACE-Uni for accurate event boundary detection
and temporal analysis, and a Captioner using InternVL-2.5 for generating
detailed object-centric descriptions. Through spatiotemporal visual prompts and
chain-of-thought reasoning, our framework generates detailed, temporally-aware
descriptions of objects' attributes, actions, statuses, interactions, and
environmental contexts without requiring additional training data. CAT-V
supports flexible user interactions through various visual prompts (points,
bounding boxes, and irregular regions) and maintains temporal sensitivity by
tracking object states and interactions across different time segments. Our
approach addresses limitations of existing video captioning methods, which
either produce overly abstract descriptions or lack object-level precision,
enabling fine-grained, object-specific descriptions while maintaining temporal
coherence and spatial accuracy. The GitHub repository for this project is
available at https://github.com/yunlong10/CAT-V
67f7cebba5c42b93f487a2b3
Yunlong Tang
claimed_verified
2025-04-10T15:45:42.596Z
6344c87f0f69ad8aa61dfcf6
https://cdn-avatars.huggingface.co/v1/production/uploads/6344c87f0f69ad8aa61dfcf6/m4QiQ_4c6GfOIZZFICy7T.jpeg
Yolo Y. Tang
user
yunlong10
67f7cebba5c42b93f487a2b4
Jing Bi
67f7cebba5c42b93f487a2b5
Chao Huang
67f7cebba5c42b93f487a2b6
Susan Liang
67f7cebba5c42b93f487a2b7
Daiki Shimada
67f7cebba5c42b93f487a2b8
Hang Hua
67f7cebba5c42b93f487a2b9
Yunzhong Xiao
67f7cebba5c42b93f487a2ba
Yizhi Song
67f7cebba5c42b93f487a2bb
Pinxin Liu
67f7cebba5c42b93f487a2bc
Mingqian Feng
67f7cebba5c42b93f487a2bd
Junjia Guo
67f7cebba5c42b93f487a2be
Zhuo Liu
67f7cebba5c42b93f487a2bf
Luchuan Song
67f7cebba5c42b93f487a2c0
Ali Vosoughi
67f7cebba5c42b93f487a2c1
Jinxi He
67f7cebba5c42b93f487a2c2
Liu He
67f7cebba5c42b93f487a2c3
Zeliang Zhang
67f7cebba5c42b93f487a2c4
Jiebo Luo
67f7cebba5c42b93f487a2c5
Chenliang Xu
Caption Anything in Video: Fine-grained Object-centric Captioning via
  Spatiotemporal Multimodal Prompting
We present CAT-V (Caption AnyThing in Video), a training-free framework for
fine-grained object-centric video captioning that enables detailed descriptions
of user-selected objects through time. CAT-V integrates three key components: a
Segmenter based on SAMURAI for precise object segmentation across frames, a
Temporal Analyzer powered by TRACE-Uni for accurate event boundary detection
and temporal analysis, and a Captioner using InternVL-2.5 for generating
detailed object-centric descriptions. Through spatiotemporal visual prompts and
chain-of-thought reasoning, our framework generates detailed, temporally-aware
descriptions of objects' attributes, actions, statuses, interactions, and
environmental contexts without requiring additional training data. CAT-V
supports flexible user interactions through various visual prompts (points,
bounding boxes, and irregular regions) and maintains temporal sensitivity by
tracking object states and interactions across different time segments. Our
approach addresses limitations of existing video captioning methods, which
either produce overly abstract descriptions or lack object-level precision,
enabling fine-grained, object-specific descriptions while maintaining temporal
coherence and spatial accuracy. The GitHub repository for this project is
available at https://github.com/yunlong10/CAT-V
CAT-V provides detailed video object-centric captions using spatiotemporal prompts without additional training, integrating segmentation, temporal analysis, and caption generation.
SAMURAI TRACE-Uni InternVL-2.5 spatiotemporal visual prompts chain-of-thought reasoning object segmentation event boundary detection temporal analysis object-centric descriptions temporal sensitivity flexible user interactions temporal coherence spatial accuracy