Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large
  Language Models in Code Generation from Scientific Plots
The remarkable progress of Multi-modal Large Language Models (MLLMs) has
attracted significant attention due to their superior performance in visual
contexts. However, their capabilities in turning visual figure to executable
code, have not been evaluated thoroughly. To address this, we introduce
Plot2Code, a comprehensive visual coding benchmark designed for a fair and
in-depth assessment of MLLMs. We carefully collect 132 manually selected
high-quality matplotlib plots across six plot types from publicly available
matplotlib galleries. For each plot, we carefully offer its source code, and an
descriptive instruction summarized by GPT-4. This approach enables Plot2Code to
extensively evaluate MLLMs' code capabilities across various input modalities.
Furthermore, we propose three automatic evaluation metrics, including code pass
rate, text-match ratio, and GPT-4V overall rating, for a fine-grained
assessment of the output code and rendered images. Instead of simply judging
pass or fail, we employ GPT-4V to make an overall judgement between the
generated and reference images, which has been shown to be consistent with
human evaluation. The evaluation results, which include analyses of 14 MLLMs
such as the proprietary GPT-4V, Gemini-Pro, and the open-sourced Mini-Gemini,
highlight the substantial challenges presented by Plot2Code. With Plot2Code, we
reveal that most existing MLLMs struggle with visual coding for text-dense
plots, heavily relying on textual instruction. We hope that the evaluation
results from Plot2Code on visual coding will guide the future development of
MLLMs. All data involved with Plot2Code are available at
https://huggingface.co/datasets/TencentARC/Plot2Code.
6642e44b273e548725c62932
Chengyue Wu
admin_assigned
2024-05-14T07:23:24.217Z
617526c9de8feb54b0ce45ad
/avatars/7faf8c6f71fc318a0113d780d376c381.svg
Wu Chengyue
user
WuChengyue
6642e44b273e548725c62933
Yixiao Ge
admin_assigned
2024-05-14T07:23:29.975Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
6642e44b273e548725c62934
Qiushan Guo
admin_assigned
2024-05-14T07:23:47.453Z
64340ef7d55dea2d0ec0de6a
/avatars/3baaa05090306910f77b368abcb27bdc.svg
qsguo
user
qiushanhku
6642e44b273e548725c62935
Jiahao Wang
admin_assigned
2024-05-14T07:25:17.531Z
65994517d3f2137415bb4d5b
https://cdn-avatars.huggingface.co/v1/production/uploads/65994517d3f2137415bb4d5b/PhjWetLTMn3bThWFLGCxR.png
WANG Jiahao
user
techmonsterwang
6642e44b273e548725c62936
Zhixuan Liang
admin_assigned
2024-05-14T07:25:26.878Z
662a471e94baa018b00c0f5c
/avatars/62a67a2ee6e4b9a7124f8b02b9b3f280.svg
Zhixuan Liang
user
Liang-ZX
6642e44b273e548725c62937
Zeyu Lu
admin_assigned
2024-05-14T07:26:03.783Z
6522ab765e7247c29172067e
/avatars/ea8d64a50d55456bd034166170a67748.svg
Zeyu Lu
user
zeyulu
6642e44b273e548725c62938
Ying Shan
6642e44b273e548725c62939
Ping Luo
Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large
  Language Models in Code Generation from Scientific Plots
The remarkable progress of Multi-modal Large Language Models (MLLMs) has
attracted significant attention due to their superior performance in visual
contexts. However, their capabilities in turning visual figure to executable
code, have not been evaluated thoroughly. To address this, we introduce
Plot2Code, a comprehensive visual coding benchmark designed for a fair and
in-depth assessment of MLLMs. We carefully collect 132 manually selected
high-quality matplotlib plots across six plot types from publicly available
matplotlib galleries. For each plot, we carefully offer its source code, and an
descriptive instruction summarized by GPT-4. This approach enables Plot2Code to
extensively evaluate MLLMs' code capabilities across various input modalities.
Furthermore, we propose three automatic evaluation metrics, including code pass
rate, text-match ratio, and GPT-4V overall rating, for a fine-grained
assessment of the output code and rendered images. Instead of simply judging
pass or fail, we employ GPT-4V to make an overall judgement between the
generated and reference images, which has been shown to be consistent with
human evaluation. The evaluation results, which include analyses of 14 MLLMs
such as the proprietary GPT-4V, Gemini-Pro, and the open-sourced Mini-Gemini,
highlight the substantial challenges presented by Plot2Code. With Plot2Code, we
reveal that most existing MLLMs struggle with visual coding for text-dense
plots, heavily relying on textual instruction. We hope that the evaluation
results from Plot2Code on visual coding will guide the future development of
MLLMs. All data involved with Plot2Code are available at
https://huggingface.co/datasets/TencentARC/Plot2Code.
Plot2Code is a visual coding benchmark for evaluating Multi-modal Large Language Models' abilities to convert visual figures into executable code, using automatic metrics for comprehensive assessment.
Multi-modal Large Language Models MLMLs Plot2Code matplotlib plots code pass rate text-match ratio GPT-4V overall rating visual coding GPT-4V Gemini-Pro Mini-Gemini