ChartCap: Mitigating Hallucination of Dense Chart Captioning
Generating accurate, informative, and hallucination-free captions for charts
remains challenging for vision language models, primarily due to the lack of
large-scale, high-quality datasets of real-world charts. However, existing
real-world chart datasets suffer from the inclusion of extraneous information
that cannot be inferred from the chart and failure to sufficiently capture
structural elements and key insights. Therefore, we introduce ChartCap, a
large-scale dataset of 565K real-world chart images paired with type-specific,
dense captions that exclude extraneous information and highlight both
structural elements and key insights in detail. To build ChartCap, we design a
four-stage pipeline that generates captions using only the discernible data
from the chart and employ a cycle consistency-based human verification, which
accelerates quality control without sacrificing accuracy. Additionally, we
propose a novel metric, the Visual Consistency Score, which evaluates caption
quality by measuring the similarity between the chart regenerated from a
caption and the original chart, independent of reference captions. Extensive
experiments confirms that models fine-tuned on ChartCap consistently generate
more accurate and informative captions with reduced hallucinations, surpassing
both open-source and proprietary models and even human-annotated captions.
6892f90b8da45ffb0a2b2510
Junyoung Lim
6892f90b8da45ffb0a2b2511
Jaewoo Ahn
6892f90b8da45ffb0a2b2512
Gunhee Kim
ChartCap: Mitigating Hallucination of Dense Chart Captioning
Generating accurate, informative, and hallucination-free captions for charts
remains challenging for vision language models, primarily due to the lack of
large-scale, high-quality datasets of real-world charts. However, existing
real-world chart datasets suffer from the inclusion of extraneous information
that cannot be inferred from the chart and failure to sufficiently capture
structural elements and key insights. Therefore, we introduce ChartCap, a
large-scale dataset of 565K real-world chart images paired with type-specific,
dense captions that exclude extraneous information and highlight both
structural elements and key insights in detail. To build ChartCap, we design a
four-stage pipeline that generates captions using only the discernible data
from the chart and employ a cycle consistency-based human verification, which
accelerates quality control without sacrificing accuracy. Additionally, we
propose a novel metric, the Visual Consistency Score, which evaluates caption
quality by measuring the similarity between the chart regenerated from a
caption and the original chart, independent of reference captions. Extensive
experiments confirms that models fine-tuned on ChartCap consistently generate
more accurate and informative captions with reduced hallucinations, surpassing
both open-source and proprietary models and even human-annotated captions.
ChartCap, a large-scale dataset with dense, type-specific captions for real-world charts, improves caption accuracy and reduces hallucinations in vision language models.
vision language models large-scale dataset real-world charts dense captions extraneous information structural elements key insights four-stage pipeline cycle consistency-based human verification Visual Consistency Score caption quality hallucinations