title: A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style
    Models on Dense Captions
summary: Curation methods for massive vision-language datasets trade off between
  dataset size and quality. However, even the highest quality of available
  curated captions are far too short to capture the rich visual detail in an
  image. To show the value of dense and highly-aligned image-text pairs, we
  collect the Densely Captioned Images (DCI) dataset, containing 8012 natural
  images human-annotated with mask-aligned descriptions averaging above 1000
  words each. With precise and reliable captions associated with specific parts
  of an image, we can evaluate vision-language models' (VLMs) understanding of
  image content with a novel task that matches each caption with its
  corresponding subcrop. As current models are often limited to 77 text tokens,
  we also introduce a summarized version (sDCI) in which each caption length is
  limited. We show that modern techniques that make progress on standard
  benchmarks do not correspond with significant improvement on our sDCI based
  benchmark. Lastly, we finetune CLIP using sDCI and show significant
  improvements over the baseline despite a small training set. By releasing the
  first human annotated dense image captioning dataset, we hope to enable the
  development of new benchmarks or fine-tuning recipes for the next generation of
  VLMs to come.
paper_authors._id: 657bc69bddc32bef8bb96956
paper_authors.name: Jack Urbanek
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:45:30.439Z
paper_authors._id: 631106397680dc699b1b1ca1
paper_authors.avatarUrl: /avatars/2c6f31b5328309cf4b42262bbb1c6372.svg
paper_authors.fullname: Jack Urbanek
paper_authors.type: user
paper_authors.user: JackUrb
paper_authors._id: 657bc69bddc32bef8bb96957
paper_authors.name: Florian Bordes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:45:41.654Z
paper_authors._id: 63166336894404e2506b8811
paper_authors.avatarUrl: /avatars/ec51a1dd86511127cf83afd4bf7c0f52.svg
paper_authors.fullname: Florian Bordes
paper_authors.type: user
paper_authors.user: Fbordes
paper_authors._id: 657bc69bddc32bef8bb96958
paper_authors.name: Pietro Astolfi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:45:48.973Z
paper_authors._id: 646388dfd4d34b01f45d87c3
paper_authors.avatarUrl: /avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
paper_authors.fullname: Pietro Astolfi
paper_authors.type: user
paper_authors.user: pietroastolfi
paper_authors._id: 657bc69bddc32bef8bb96959
paper_authors.name: Mary Williamson
paper_authors._id: 657bc69bddc32bef8bb9695a
paper_authors.name: Vasu Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T08:22:27.973Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 657bc69bddc32bef8bb9695b
paper_authors.name: Adriana Romero-Soriano
paper_title: A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style
    Models on Dense Captions
paper_summary: Curation methods for massive vision-language datasets trade off between
  dataset size and quality. However, even the highest quality of available
  curated captions are far too short to capture the rich visual detail in an
  image. To show the value of dense and highly-aligned image-text pairs, we
  collect the Densely Captioned Images (DCI) dataset, containing 8012 natural
  images human-annotated with mask-aligned descriptions averaging above 1000
  words each. With precise and reliable captions associated with specific parts
  of an image, we can evaluate vision-language models' (VLMs) understanding of
  image content with a novel task that matches each caption with its
  corresponding subcrop. As current models are often limited to 77 text tokens,
  we also introduce a summarized version (sDCI) in which each caption length is
  limited. We show that modern techniques that make progress on standard
  benchmarks do not correspond with significant improvement on our sDCI based
  benchmark. Lastly, we finetune CLIP using sDCI and show significant
  improvements over the baseline despite a small training set. By releasing the
  first human annotated dense image captioning dataset, we hope to enable the
  development of new benchmarks or fine-tuning recipes for the next generation of
  VLMs to come.
paper_ai_summary: A human-annotated dataset with dense captions significantly improves the evaluation and fine-tuning of vision-language models beyond standard benchmarks.
paper_ai_keywords: Vision-language models (VLMs) mask-aligned descriptions subcrop matching text tokens densel captioning CLIP fine-tuning