title: FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
summary: Multimodal Large Language Models (MLLMs) have experienced rapid development
  in recent years. However, in the financial domain, there is a notable lack of
  effective and specialized multimodal evaluation datasets. To advance the
  development of MLLMs in the finance domain, we introduce FinMME, encompassing
  more than 11,000 high-quality financial research samples across 18 financial
  domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We
  ensure data quality through 20 annotators and carefully designed validation
  mechanisms. Additionally, we develop FinScore, an evaluation system
  incorporating hallucination penalties and multi-dimensional capability
  assessment to provide an unbiased evaluation. Extensive experimental results
  demonstrate that even state-of-the-art models like GPT-4o exhibit
  unsatisfactory performance on FinMME, highlighting its challenging nature. The
  benchmark exhibits high robustness with prediction variations under different
  prompts remaining below 1%, demonstrating superior reliability compared to
  existing datasets. Our dataset and evaluation protocol are available at
  https://huggingface.co/datasets/luojunyu/FinMME and
  https://github.com/luo-junyu/FinMME.
paper_authors._id: 683d04c751706d12b2c262ea
paper_authors.name: Junyu Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:26.483Z
paper_authors._id: 642da1cd99f3110ac27caca5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642da1cd99f3110ac27caca5/C1QJY3R_ZdaeANG1y8iW7.jpeg
paper_authors.fullname: junyu
paper_authors.type: user
paper_authors.user: luojunyu
paper_authors._id: 683d04c751706d12b2c262eb
paper_authors.name: Zhizhuo Kou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:22.133Z
paper_authors._id: 65a9c8652bf3e0cbbfcab2c8
paper_authors.avatarUrl: /avatars/fc690a78b5f2e94e08a40059ae40625c.svg
paper_authors.fullname: Alan KOU
paper_authors.type: user
paper_authors.user: alan1027
paper_authors._id: 683d04c751706d12b2c262ec
paper_authors.name: Liming Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:24.346Z
paper_authors._id: 6434c115a5aed21dd11981c5
paper_authors.avatarUrl: /avatars/d51e6e384cfc3affe578e7816bcebb35.svg
paper_authors.fullname: Yang Liming
paper_authors.type: user
paper_authors.user: chunfenri
paper_authors._id: 683d04c751706d12b2c262ed
paper_authors.name: Xiao Luo
paper_authors._id: 683d04c751706d12b2c262ee
paper_authors.name: Jinsheng Huang
paper_authors._id: 683d04c751706d12b2c262ef
paper_authors.name: Zhiping Xiao
paper_authors._id: 683d04c751706d12b2c262f0
paper_authors.name: Jingshu Peng
paper_authors._id: 683d04c751706d12b2c262f1
paper_authors.name: Chengzhong Liu
paper_authors._id: 683d04c751706d12b2c262f2
paper_authors.name: Jiaming Ji
paper_authors._id: 683d04c751706d12b2c262f3
paper_authors.name: Xuanzhe Liu
paper_authors._id: 683d04c751706d12b2c262f4
paper_authors.name: Sirui Han
paper_authors._id: 683d04c751706d12b2c262f5
paper_authors.name: Ming Zhang
paper_authors._id: 683d04c751706d12b2c262f6
paper_authors.name: Yike Guo
paper_title: FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
paper_summary: Multimodal Large Language Models (MLLMs) have experienced rapid development
  in recent years. However, in the financial domain, there is a notable lack of
  effective and specialized multimodal evaluation datasets. To advance the
  development of MLLMs in the finance domain, we introduce FinMME, encompassing
  more than 11,000 high-quality financial research samples across 18 financial
  domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We
  ensure data quality through 20 annotators and carefully designed validation
  mechanisms. Additionally, we develop FinScore, an evaluation system
  incorporating hallucination penalties and multi-dimensional capability
  assessment to provide an unbiased evaluation. Extensive experimental results
  demonstrate that even state-of-the-art models like GPT-4o exhibit
  unsatisfactory performance on FinMME, highlighting its challenging nature. The
  benchmark exhibits high robustness with prediction variations under different
  prompts remaining below 1%, demonstrating superior reliability compared to
  existing datasets. Our dataset and evaluation protocol are available at
  https://huggingface.co/datasets/luojunyu/FinMME and
  https://github.com/luo-junyu/FinMME.
paper_ai_summary: FinMME is a comprehensive multimodal dataset for financial research and FinScore is an evaluation system that highlights the challenges faced by even advanced models like GPT-4o in the finance domain.
paper_ai_keywords: Multimodal Large Language Models FinMME financial research samples high-quality dataset financial domains asset classes chart types data quality FinScore hallucination penalties multi-dimensional capability assessment benchmark dataset prediction robustness