ChartLens: Fine-grained Visual Attribution in Charts
The growing capabilities of multimodal large language models (MLLMs) have
advanced tasks like chart understanding. However, these models often suffer
from hallucinations, where generated text sequences conflict with the provided
visual data. To address this, we introduce Post-Hoc Visual Attribution for
Charts, which identifies fine-grained chart elements that validate a given
chart-associated response. We propose ChartLens, a novel chart attribution
algorithm that uses segmentation-based techniques to identify chart objects and
employs set-of-marks prompting with MLLMs for fine-grained visual attribution.
Additionally, we present ChartVA-Eval, a benchmark with synthetic and
real-world charts from diverse domains like finance, policy, and economics,
featuring fine-grained attribution annotations. Our evaluations show that
ChartLens improves fine-grained attributions by 26-66%.
68392279896eb9ceb71fac39
Manan Suri
68392279896eb9ceb71fac3a
Puneet Mathur
extracted_pending
2025-05-30T03:14:02.414Z
65c16444d4c3b8dff2f0d78d
/avatars/4ed764c1657bd260d2a12ba61c111062.svg
Puneet Mathur
user
puneetm
68392279896eb9ceb71fac3b
Nedim Lipka
68392279896eb9ceb71fac3c
Franck Dernoncourt
claimed_verified
2025-05-30T06:53:28.558Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
68392279896eb9ceb71fac3d
Ryan A. Rossi
68392279896eb9ceb71fac3e
Dinesh Manocha
ChartLens: Fine-grained Visual Attribution in Charts
The growing capabilities of multimodal large language models (MLLMs) have
advanced tasks like chart understanding. However, these models often suffer
from hallucinations, where generated text sequences conflict with the provided
visual data. To address this, we introduce Post-Hoc Visual Attribution for
Charts, which identifies fine-grained chart elements that validate a given
chart-associated response. We propose ChartLens, a novel chart attribution
algorithm that uses segmentation-based techniques to identify chart objects and
employs set-of-marks prompting with MLLMs for fine-grained visual attribution.
Additionally, we present ChartVA-Eval, a benchmark with synthetic and
real-world charts from diverse domains like finance, policy, and economics,
featuring fine-grained attribution annotations. Our evaluations show that
ChartLens improves fine-grained attributions by 26-66%.
ChartLens enhances multimodal language models with fine-grained visual attributions, improving the accuracy of chart understanding by 26-66%.
multimodal large language models MLLMs chart understanding hallucinations Post-Hoc Visual Attribution ChartLens segmentation-based techniques set-of-marks prompting ChartVA-Eval synthetic charts real-world charts fine-grained attribution annotations