title: COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for
    Fine-Grained Understanding and Generation
summary: This paper introduces the COCONut-PanCap dataset, created to enhance panoptic
  segmentation and grounded image captioning. Building upon the COCO dataset with
  advanced COCONut panoptic masks, this dataset aims to overcome limitations in
  existing image-text datasets that often lack detailed, scene-comprehensive
  descriptions. The COCONut-PanCap dataset incorporates fine-grained,
  region-level captions grounded in panoptic segmentation masks, ensuring
  consistency and improving the detail of generated captions. Through
  human-edited, densely annotated descriptions, COCONut-PanCap supports improved
  training of vision-language models (VLMs) for image understanding and
  generative models for text-to-image tasks. Experimental results demonstrate
  that COCONut-PanCap significantly boosts performance across understanding and
  generation tasks, offering complementary benefits to large-scale datasets. This
  dataset sets a new benchmark for evaluating models on joint panoptic
  segmentation and grounded captioning tasks, addressing the need for
  high-quality, detailed image-text annotations in multi-modal learning.
paper_authors._id: 67a3ad7447edcbb9e1f1e2f0
paper_authors.name: Xueqing Deng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-05T18:27:02.349Z
paper_authors._id: 65ca9b1743207e438a95e90c
paper_authors.avatarUrl: /avatars/8f7bde1c44d8e665a29ee08ce7fedfa4.svg
paper_authors.fullname: Xueqing Deng
paper_authors.type: user
paper_authors.user: xdeng77
paper_authors._id: 67a3ad7447edcbb9e1f1e2f1
paper_authors.name: Qihang Yu
paper_authors._id: 67a3ad7447edcbb9e1f1e2f2
paper_authors.name: Ali Athar
paper_authors._id: 67a3ad7447edcbb9e1f1e2f3
paper_authors.name: Chenglin Yang
paper_authors._id: 67a3ad7447edcbb9e1f1e2f4
paper_authors.name: Linjie Yang
paper_authors._id: 67a3ad7447edcbb9e1f1e2f5
paper_authors.name: Xiaojie Jin
paper_authors._id: 67a3ad7447edcbb9e1f1e2f6
paper_authors.name: Xiaohui Shen
paper_authors._id: 67a3ad7447edcbb9e1f1e2f7
paper_authors.name: Liang-Chieh Chen
paper_title: COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for
    Fine-Grained Understanding and Generation
paper_summary: This paper introduces the COCONut-PanCap dataset, created to enhance panoptic
  segmentation and grounded image captioning. Building upon the COCO dataset with
  advanced COCONut panoptic masks, this dataset aims to overcome limitations in
  existing image-text datasets that often lack detailed, scene-comprehensive
  descriptions. The COCONut-PanCap dataset incorporates fine-grained,
  region-level captions grounded in panoptic segmentation masks, ensuring
  consistency and improving the detail of generated captions. Through
  human-edited, densely annotated descriptions, COCONut-PanCap supports improved
  training of vision-language models (VLMs) for image understanding and
  generative models for text-to-image tasks. Experimental results demonstrate
  that COCONut-PanCap significantly boosts performance across understanding and
  generation tasks, offering complementary benefits to large-scale datasets. This
  dataset sets a new benchmark for evaluating models on joint panoptic
  segmentation and grounded captioning tasks, addressing the need for
  high-quality, detailed image-text annotations in multi-modal learning.
paper_ai_summary: The COCONut-PanCap dataset integrates detailed panoptic masks and region-level captions to improve vision-language models and generative models for better performance in image understanding and text-to-image tasks.
paper_ai_keywords: panoptic segmentation grounded image captioning COCONut dataset VLMs vision-language models text-to-image tasks human-edited annotations image-text annotations multi-modal learning