title: Describe Anything: Detailed Localized Image and Video Captioning
summary: Generating detailed and accurate descriptions for specific regions in images
  and videos remains a fundamental challenge for vision-language models. We
  introduce the Describe Anything Model (DAM), a model designed for detailed
  localized captioning (DLC). DAM preserves both local details and global context
  through two key innovations: a focal prompt, which ensures high-resolution
  encoding of targeted regions, and a localized vision backbone, which integrates
  precise localization with its broader context. To tackle the scarcity of
  high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data
  Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and
  expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark
  designed to evaluate DLC without relying on reference captions. DAM sets new
  state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and
  detailed multi-sentence localized image and video captioning.
paper_authors._id: 6808467a867c3ef14f8326ce
paper_authors.name: Long Lian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:28:14.686Z
paper_authors._id: 63797c273f575acc2f6893c0
paper_authors.avatarUrl: /avatars/32d7a6a8881c8c4d80a097b732ed24b6.svg
paper_authors.fullname: Long(Tony) Lian
paper_authors.type: user
paper_authors.user: longlian
paper_authors._id: 6808467a867c3ef14f8326cf
paper_authors.name: Yifan Ding
paper_authors._id: 6808467a867c3ef14f8326d0
paper_authors.name: Yunhao Ge
paper_authors._id: 6808467a867c3ef14f8326d1
paper_authors.name: Sifei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:46:08.248Z
paper_authors._id: 62fab69f8cd542e895bafd6e
paper_authors.avatarUrl: /avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
paper_authors.fullname: Sifei Liu
paper_authors.type: user
paper_authors.user: zwrq
paper_authors._id: 6808467a867c3ef14f8326d2
paper_authors.name: Hanzi Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:46:14.917Z
paper_authors._id: 645bd14978730bcc10414267
paper_authors.avatarUrl: /avatars/155cefe7bd5878532ce19693be5a13ec.svg
paper_authors.fullname: Hanzi Mao
paper_authors.type: user
paper_authors.user: hannamao
paper_authors._id: 6808467a867c3ef14f8326d3
paper_authors.name: Boyi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:28:09.738Z
paper_authors._id: 620dd3888528f797e88cb9b5
paper_authors.avatarUrl: /avatars/af04728788d78fe7d6375e19e32a535e.svg
paper_authors.fullname: Boyi Li
paper_authors.type: user
paper_authors.user: Boyiliee
paper_authors._id: 6808467a867c3ef14f8326d4
paper_authors.name: Marco Pavone
paper_authors._id: 6808467a867c3ef14f8326d5
paper_authors.name: Ming-Yu Liu
paper_authors._id: 6808467a867c3ef14f8326d6
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:46:25.340Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_authors._id: 6808467a867c3ef14f8326d7
paper_authors.name: Adam Yala
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:28:12.415Z
paper_authors._id: 6333a9195a032dcd095dda13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664329996201-noauth.jpeg
paper_authors.fullname: Adam Yala
paper_authors.type: user
paper_authors.user: yala
paper_authors._id: 6808467a867c3ef14f8326d8
paper_authors.name: Yin Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:28:06.739Z
paper_authors._id: 649f05367b57fab3a5b27c8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
paper_authors.fullname: Yin Cui
paper_authors.type: user
paper_authors.user: richardaecn
paper_title: Describe Anything: Detailed Localized Image and Video Captioning
paper_summary: Generating detailed and accurate descriptions for specific regions in images
  and videos remains a fundamental challenge for vision-language models. We
  introduce the Describe Anything Model (DAM), a model designed for detailed
  localized captioning (DLC). DAM preserves both local details and global context
  through two key innovations: a focal prompt, which ensures high-resolution
  encoding of targeted regions, and a localized vision backbone, which integrates
  precise localization with its broader context. To tackle the scarcity of
  high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data
  Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and
  expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark
  designed to evaluate DLC without relying on reference captions. DAM sets new
  state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and
  detailed multi-sentence localized image and video captioning.
paper_ai_summary: The Describe Anything Model (DAM) leverages a focal prompt and localized vision backbone to achieve detailed localized captioning, outperforming existing models on various benchmarks through a semi-supervised data pipeline.
paper_ai_keywords: focal prompt localized vision backbone detailed localized captioning semi-supervised learning DLC-SDP DLC-Bench