COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for
  Fine-Grained Understanding and Generation
This paper introduces the COCONut-PanCap dataset, created to enhance panoptic
segmentation and grounded image captioning. Building upon the COCO dataset with
advanced COCONut panoptic masks, this dataset aims to overcome limitations in
existing image-text datasets that often lack detailed, scene-comprehensive
descriptions. The COCONut-PanCap dataset incorporates fine-grained,
region-level captions grounded in panoptic segmentation masks, ensuring
consistency and improving the detail of generated captions. Through
human-edited, densely annotated descriptions, COCONut-PanCap supports improved
training of vision-language models (VLMs) for image understanding and
generative models for text-to-image tasks. Experimental results demonstrate
that COCONut-PanCap significantly boosts performance across understanding and
generation tasks, offering complementary benefits to large-scale datasets. This
dataset sets a new benchmark for evaluating models on joint panoptic
segmentation and grounded captioning tasks, addressing the need for
high-quality, detailed image-text annotations in multi-modal learning.
67a3ad7447edcbb9e1f1e2f0
Xueqing Deng
extracted_pending
2025-02-05T18:27:02.349Z
65ca9b1743207e438a95e90c
/avatars/8f7bde1c44d8e665a29ee08ce7fedfa4.svg
Xueqing Deng
user
xdeng77
67a3ad7447edcbb9e1f1e2f1
Qihang Yu
67a3ad7447edcbb9e1f1e2f2
Ali Athar
67a3ad7447edcbb9e1f1e2f3
Chenglin Yang
67a3ad7447edcbb9e1f1e2f4
Linjie Yang
67a3ad7447edcbb9e1f1e2f5
Xiaojie Jin
67a3ad7447edcbb9e1f1e2f6
Xiaohui Shen
67a3ad7447edcbb9e1f1e2f7
Liang-Chieh Chen
COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for
  Fine-Grained Understanding and Generation
This paper introduces the COCONut-PanCap dataset, created to enhance panoptic
segmentation and grounded image captioning. Building upon the COCO dataset with
advanced COCONut panoptic masks, this dataset aims to overcome limitations in
existing image-text datasets that often lack detailed, scene-comprehensive
descriptions. The COCONut-PanCap dataset incorporates fine-grained,
region-level captions grounded in panoptic segmentation masks, ensuring
consistency and improving the detail of generated captions. Through
human-edited, densely annotated descriptions, COCONut-PanCap supports improved
training of vision-language models (VLMs) for image understanding and
generative models for text-to-image tasks. Experimental results demonstrate
that COCONut-PanCap significantly boosts performance across understanding and
generation tasks, offering complementary benefits to large-scale datasets. This
dataset sets a new benchmark for evaluating models on joint panoptic
segmentation and grounded captioning tasks, addressing the need for
high-quality, detailed image-text annotations in multi-modal learning.
The COCONut-PanCap dataset integrates detailed panoptic masks and region-level captions to improve vision-language models and generative models for better performance in image understanding and text-to-image tasks.
panoptic segmentation grounded image captioning COCONut dataset VLMs vision-language models text-to-image tasks human-edited annotations image-text annotations multi-modal learning