title: ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via
    Chart-to-Code Generation
summary: We introduce a new benchmark, ChartMimic, aimed at assessing the
  visually-grounded code generation capabilities of large multimodal models
  (LMMs). ChartMimic utilizes information-intensive visual charts and textual
  instructions as inputs, requiring LMMs to generate the corresponding code for
  chart rendering. ChartMimic includes 1,000 human-curated (figure, instruction,
  code) triplets, which represent the authentic chart use cases found in
  scientific papers across various domains(e.g., Physics, Computer Science,
  Economics, etc). These charts span 18 regular types and 4 advanced types,
  diversifying into 191 subcategories. Furthermore, we propose multi-level
  evaluation metrics to provide an automatic and thorough assessment of the
  output code and the rendered charts. Unlike existing code generation
  benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to
  harmonize a blend of cognitive capabilities, encompassing visual understanding,
  code generation, and cross-modal reasoning. The evaluation of 3 proprietary
  models and 11 open-weight models highlights the substantial challenges posed by
  ChartMimic. Even the advanced GPT-4V, Claude-3-opus only achieve an average
  score of 73.2 and 53.7, respectively, indicating significant room for
  improvement. We anticipate that ChartMimic will inspire the development of
  LMMs, advancing the pursuit of artificial general intelligence.
paper_authors._id: 666fdf3bc16d62cbb045cae0
paper_authors.name: Chufan Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:01:12.424Z
paper_authors._id: 6373b559c3b104ef179cf503
paper_authors.avatarUrl: /avatars/36f36eea72a100ee56b37c6d185b47c0.svg
paper_authors.fullname: Shi
paper_authors.type: user
paper_authors.user: Chufan
paper_authors._id: 666fdf3bc16d62cbb045cae1
paper_authors.name: Cheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:31:03.951Z
paper_authors._id: 64b75db417570fdff9b43fd9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b75db417570fdff9b43fd9/-NcNL_QzuiAAvsBIQG3xo.png
paper_authors.fullname: Cheng YANG
paper_authors.type: user
paper_authors.user: Cheng-YANG
paper_authors._id: 666fdf3bc16d62cbb045cae2
paper_authors.name: Yaxin Liu
paper_authors._id: 666fdf3bc16d62cbb045cae3
paper_authors.name: Bo Shui
paper_authors._id: 666fdf3bc16d62cbb045cae4
paper_authors.name: Junjie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:20:00.036Z
paper_authors._id: 62579c55b98dcaa7e0de285d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
paper_authors.fullname: wangjunjie
paper_authors.type: user
paper_authors.user: wanng
paper_authors._id: 666fdf3bc16d62cbb045cae5
paper_authors.name: Mohan Jing
paper_authors._id: 666fdf3bc16d62cbb045cae6
paper_authors.name: Linran Xu
paper_authors._id: 666fdf3bc16d62cbb045cae7
paper_authors.name: Xinyu Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:19.706Z
paper_authors._id: 623b290048f658f28aef79f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648044277149-noauth.jpeg
paper_authors.fullname: Xinyu Zhu
paper_authors.type: user
paper_authors.user: TianHongZXY
paper_authors._id: 666fdf3bc16d62cbb045cae8
paper_authors.name: Siheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:09:59.455Z
paper_authors._id: 62f71b23d3bdacb7eec43b1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f71b23d3bdacb7eec43b1f/6RNVG9zug9tutjo5i8T4C.jpeg
paper_authors.fullname: SihengLi
paper_authors.type: user
paper_authors.user: SihengLi
paper_authors._id: 666fdf3bc16d62cbb045cae9
paper_authors.name: Yuxiang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:12:32.284Z
paper_authors._id: 6239386535384c2bcccd2a4f
paper_authors.avatarUrl: /avatars/c62cefc6235210ca12b46beb62751d8b.svg
paper_authors.fullname: Yuxiang Zhang
paper_authors.type: user
paper_authors.user: Joelzhang
paper_authors._id: 666fdf3bc16d62cbb045caea
paper_authors.name: Gongye Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:10:43.252Z
paper_authors._id: 6553316bf151de82f6a23e1d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
paper_authors.fullname: Gongye Liu
paper_authors.type: user
paper_authors.user: liuhuohuo
paper_authors._id: 666fdf3bc16d62cbb045caeb
paper_authors.name: Xiaomei Nie
paper_authors._id: 666fdf3bc16d62cbb045caec
paper_authors.name: Deng Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:34:42.506Z
paper_authors._id: 646735a1946476c5d214e5b2
paper_authors.avatarUrl: /avatars/23392fda23fa6288fcb309f7eaa73472.svg
paper_authors.fullname: Deng Cai
paper_authors.type: user
paper_authors.user: jcyk
paper_authors._id: 666fdf3bc16d62cbb045caed
paper_authors.name: Yujiu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:11:00.402Z
paper_authors._id: 64ca1fe838837b12d5e529b7
paper_authors.avatarUrl: /avatars/44a3ad9e59318784ac531993b5f69f6b.svg
paper_authors.fullname: Yujiu Yang
paper_authors.type: user
paper_authors.user: Thu-redrobot
paper_title: ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via
    Chart-to-Code Generation
paper_summary: We introduce a new benchmark, ChartMimic, aimed at assessing the
  visually-grounded code generation capabilities of large multimodal models
  (LMMs). ChartMimic utilizes information-intensive visual charts and textual
  instructions as inputs, requiring LMMs to generate the corresponding code for
  chart rendering. ChartMimic includes 1,000 human-curated (figure, instruction,
  code) triplets, which represent the authentic chart use cases found in
  scientific papers across various domains(e.g., Physics, Computer Science,
  Economics, etc). These charts span 18 regular types and 4 advanced types,
  diversifying into 191 subcategories. Furthermore, we propose multi-level
  evaluation metrics to provide an automatic and thorough assessment of the
  output code and the rendered charts. Unlike existing code generation
  benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to
  harmonize a blend of cognitive capabilities, encompassing visual understanding,
  code generation, and cross-modal reasoning. The evaluation of 3 proprietary
  models and 11 open-weight models highlights the substantial challenges posed by
  ChartMimic. Even the advanced GPT-4V, Claude-3-opus only achieve an average
  score of 73.2 and 53.7, respectively, indicating significant room for
  improvement. We anticipate that ChartMimic will inspire the development of
  LMMs, advancing the pursuit of artificial general intelligence.
paper_ai_summary: ChartMimic is a new benchmark assessing large multimodal models' visually-grounded code generation for chart rendering, evaluating multi-level cognitive capabilities with diverse chart types and metrics.
paper_ai_keywords: multimodal models visually-grounded code generation visual charts textual instructions multi-level evaluation metrics cognitive capabilities visual understanding cross-modal reasoning