Describe Anything: Detailed Localized Image and Video Captioning
Generating detailed and accurate descriptions for specific regions in images
and videos remains a fundamental challenge for vision-language models. We
introduce the Describe Anything Model (DAM), a model designed for detailed
localized captioning (DLC). DAM preserves both local details and global context
through two key innovations: a focal prompt, which ensures high-resolution
encoding of targeted regions, and a localized vision backbone, which integrates
precise localization with its broader context. To tackle the scarcity of
high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data
Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and
expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark
designed to evaluate DLC without relying on reference captions. DAM sets new
state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and
detailed multi-sentence localized image and video captioning.
6808467a867c3ef14f8326ce
Long Lian
claimed_verified
2025-04-23T08:28:14.686Z
63797c273f575acc2f6893c0
/avatars/32d7a6a8881c8c4d80a097b732ed24b6.svg
Long(Tony) Lian
user
longlian
6808467a867c3ef14f8326cf
Yifan Ding
6808467a867c3ef14f8326d0
Yunhao Ge
6808467a867c3ef14f8326d1
Sifei Liu
admin_assigned
2025-04-23T13:46:08.248Z
62fab69f8cd542e895bafd6e
/avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
Sifei Liu
user
zwrq
6808467a867c3ef14f8326d2
Hanzi Mao
admin_assigned
2025-04-23T13:46:14.917Z
645bd14978730bcc10414267
/avatars/155cefe7bd5878532ce19693be5a13ec.svg
Hanzi Mao
user
hannamao
6808467a867c3ef14f8326d3
Boyi Li
claimed_verified
2025-04-23T08:28:09.738Z
620dd3888528f797e88cb9b5
/avatars/af04728788d78fe7d6375e19e32a535e.svg
Boyi Li
user
Boyiliee
6808467a867c3ef14f8326d4
Marco Pavone
6808467a867c3ef14f8326d5
Ming-Yu Liu
6808467a867c3ef14f8326d6
Trevor Darrell
admin_assigned
2025-04-23T13:46:25.340Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
6808467a867c3ef14f8326d7
Adam Yala
claimed_verified
2025-04-23T08:28:12.415Z
6333a9195a032dcd095dda13
https://cdn-avatars.huggingface.co/v1/production/uploads/1664329996201-noauth.jpeg
Adam Yala
user
yala
6808467a867c3ef14f8326d8
Yin Cui
claimed_verified
2025-04-23T08:28:06.739Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
Describe Anything: Detailed Localized Image and Video Captioning
Generating detailed and accurate descriptions for specific regions in images
and videos remains a fundamental challenge for vision-language models. We
introduce the Describe Anything Model (DAM), a model designed for detailed
localized captioning (DLC). DAM preserves both local details and global context
through two key innovations: a focal prompt, which ensures high-resolution
encoding of targeted regions, and a localized vision backbone, which integrates
precise localization with its broader context. To tackle the scarcity of
high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data
Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and
expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark
designed to evaluate DLC without relying on reference captions. DAM sets new
state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and
detailed multi-sentence localized image and video captioning.
The Describe Anything Model (DAM) leverages a focal prompt and localized vision backbone to achieve detailed localized captioning, outperforming existing models on various benchmarks through a semi-supervised data pipeline.
focal prompt localized vision backbone detailed localized captioning semi-supervised learning DLC-SDP DLC-Bench