title: Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large
    Language Models in Code Generation from Scientific Plots
summary: The remarkable progress of Multi-modal Large Language Models (MLLMs) has
  attracted significant attention due to their superior performance in visual
  contexts. However, their capabilities in turning visual figure to executable
  code, have not been evaluated thoroughly. To address this, we introduce
  Plot2Code, a comprehensive visual coding benchmark designed for a fair and
  in-depth assessment of MLLMs. We carefully collect 132 manually selected
  high-quality matplotlib plots across six plot types from publicly available
  matplotlib galleries. For each plot, we carefully offer its source code, and an
  descriptive instruction summarized by GPT-4. This approach enables Plot2Code to
  extensively evaluate MLLMs' code capabilities across various input modalities.
  Furthermore, we propose three automatic evaluation metrics, including code pass
  rate, text-match ratio, and GPT-4V overall rating, for a fine-grained
  assessment of the output code and rendered images. Instead of simply judging
  pass or fail, we employ GPT-4V to make an overall judgement between the
  generated and reference images, which has been shown to be consistent with
  human evaluation. The evaluation results, which include analyses of 14 MLLMs
  such as the proprietary GPT-4V, Gemini-Pro, and the open-sourced Mini-Gemini,
  highlight the substantial challenges presented by Plot2Code. With Plot2Code, we
  reveal that most existing MLLMs struggle with visual coding for text-dense
  plots, heavily relying on textual instruction. We hope that the evaluation
  results from Plot2Code on visual coding will guide the future development of
  MLLMs. All data involved with Plot2Code are available at
  https://huggingface.co/datasets/TencentARC/Plot2Code.
paper_authors._id: 6642e44b273e548725c62932
paper_authors.name: Chengyue Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:23:24.217Z
paper_authors._id: 617526c9de8feb54b0ce45ad
paper_authors.avatarUrl: /avatars/7faf8c6f71fc318a0113d780d376c381.svg
paper_authors.fullname: Wu Chengyue
paper_authors.type: user
paper_authors.user: WuChengyue
paper_authors._id: 6642e44b273e548725c62933
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:23:29.975Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 6642e44b273e548725c62934
paper_authors.name: Qiushan Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:23:47.453Z
paper_authors._id: 64340ef7d55dea2d0ec0de6a
paper_authors.avatarUrl: /avatars/3baaa05090306910f77b368abcb27bdc.svg
paper_authors.fullname: qsguo
paper_authors.type: user
paper_authors.user: qiushanhku
paper_authors._id: 6642e44b273e548725c62935
paper_authors.name: Jiahao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:25:17.531Z
paper_authors._id: 65994517d3f2137415bb4d5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65994517d3f2137415bb4d5b/PhjWetLTMn3bThWFLGCxR.png
paper_authors.fullname: WANG Jiahao
paper_authors.type: user
paper_authors.user: techmonsterwang
paper_authors._id: 6642e44b273e548725c62936
paper_authors.name: Zhixuan Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:25:26.878Z
paper_authors._id: 662a471e94baa018b00c0f5c
paper_authors.avatarUrl: /avatars/62a67a2ee6e4b9a7124f8b02b9b3f280.svg
paper_authors.fullname: Zhixuan Liang
paper_authors.type: user
paper_authors.user: Liang-ZX
paper_authors._id: 6642e44b273e548725c62937
paper_authors.name: Zeyu Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:26:03.783Z
paper_authors._id: 6522ab765e7247c29172067e
paper_authors.avatarUrl: /avatars/ea8d64a50d55456bd034166170a67748.svg
paper_authors.fullname: Zeyu Lu
paper_authors.type: user
paper_authors.user: zeyulu
paper_authors._id: 6642e44b273e548725c62938
paper_authors.name: Ying Shan
paper_authors._id: 6642e44b273e548725c62939
paper_authors.name: Ping Luo
paper_title: Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large
    Language Models in Code Generation from Scientific Plots
paper_summary: The remarkable progress of Multi-modal Large Language Models (MLLMs) has
  attracted significant attention due to their superior performance in visual
  contexts. However, their capabilities in turning visual figure to executable
  code, have not been evaluated thoroughly. To address this, we introduce
  Plot2Code, a comprehensive visual coding benchmark designed for a fair and
  in-depth assessment of MLLMs. We carefully collect 132 manually selected
  high-quality matplotlib plots across six plot types from publicly available
  matplotlib galleries. For each plot, we carefully offer its source code, and an
  descriptive instruction summarized by GPT-4. This approach enables Plot2Code to
  extensively evaluate MLLMs' code capabilities across various input modalities.
  Furthermore, we propose three automatic evaluation metrics, including code pass
  rate, text-match ratio, and GPT-4V overall rating, for a fine-grained
  assessment of the output code and rendered images. Instead of simply judging
  pass or fail, we employ GPT-4V to make an overall judgement between the
  generated and reference images, which has been shown to be consistent with
  human evaluation. The evaluation results, which include analyses of 14 MLLMs
  such as the proprietary GPT-4V, Gemini-Pro, and the open-sourced Mini-Gemini,
  highlight the substantial challenges presented by Plot2Code. With Plot2Code, we
  reveal that most existing MLLMs struggle with visual coding for text-dense
  plots, heavily relying on textual instruction. We hope that the evaluation
  results from Plot2Code on visual coding will guide the future development of
  MLLMs. All data involved with Plot2Code are available at
  https://huggingface.co/datasets/TencentARC/Plot2Code.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) Plot2Code matplotlib plots source code GPT-4 GPT-4V code pass rate text-match ratio GPT-4V overall rating Mini-Gemini