title: BizFinBench: A Business-Driven Real-World Financial Benchmark for
    Evaluating LLMs
summary: Large language models excel in general tasks, yet assessing their reliability
  in logic-heavy, precision-critical domains like finance, law, and healthcare
  remains challenging. To address this, we introduce BizFinBench, the first
  benchmark specifically designed to evaluate LLMs in real-world financial
  applications. BizFinBench consists of 6,781 well-annotated queries in Chinese,
  spanning five dimensions: numerical calculation, reasoning, information
  extraction, prediction recognition, and knowledge-based question answering,
  grouped into nine fine-grained categories. The benchmark includes both
  objective and subjective metrics. We also introduce IteraJudge, a novel LLM
  evaluation method that reduces bias when LLMs serve as evaluators in objective
  metrics. We benchmark 25 models, including both proprietary and open-source
  systems. Extensive experiments show that no model dominates across all tasks.
  Our evaluation reveals distinct capability patterns: (1) In Numerical
  Calculation, Claude-3.5-Sonnet (63.18) and DeepSeek-R1 (64.04) lead, while
  smaller models like Qwen2.5-VL-3B (15.92) lag significantly; (2) In Reasoning,
  proprietary models dominate (ChatGPT-o3: 83.58, Gemini-2.0-Flash: 81.15), with
  open-source models trailing by up to 19.49 points; (3) In Information
  Extraction, the performance spread is the largest, with DeepSeek-R1 scoring
  71.46, while Qwen3-1.7B scores 11.23; (4) In Prediction Recognition,
  performance variance is minimal, with top models scoring between 39.16 and
  50.00. We find that while current LLMs handle routine finance queries
  competently, they struggle with complex scenarios requiring cross-concept
  reasoning. BizFinBench offers a rigorous, business-aligned benchmark for future
  research. The code and dataset are available at
  https://github.com/HiThink-Research/BizFinBench.
paper_authors._id: 683536ec70d215849adfc236
paper_authors.name: Guilong Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:23:29.797Z
paper_authors._id: 6440f70f1a80f6d83cadfd16
paper_authors.avatarUrl: /avatars/04790922837dac81747e80bd0ee0a1cf.svg
paper_authors.fullname: luguilong
paper_authors.type: user
paper_authors.user: guilong
paper_authors._id: 683536ec70d215849adfc237
paper_authors.name: Xuntao Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:23:39.616Z
paper_authors._id: 672b138db4215fd3888e0a8f
paper_authors.avatarUrl: /avatars/e90fe671a1db66401db88429fae9a763.svg
paper_authors.fullname: guo
paper_authors.type: user
paper_authors.user: xuntao
paper_authors._id: 683536ec70d215849adfc238
paper_authors.name: Rongjunchen Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-27T03:52:17.018Z
paper_authors._id: 6555df426947208b7741b637
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6555df426947208b7741b637/b7ply-HyaPKXrPvRNh21K.jpeg
paper_authors.fullname: Rongjunchen Zhang
paper_authors.type: user
paper_authors.user: Tinker250
paper_authors._id: 683536ec70d215849adfc239
paper_authors.name: Wenqiao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:49:28.216Z
paper_authors._id: 648add6aff6123185eb185a8
paper_authors.avatarUrl: /avatars/e37dfa680c1bb86c721165f03eb79e97.svg
paper_authors.fullname: WNQzhu
paper_authors.type: user
paper_authors.user: Qlisp
paper_authors._id: 683536ec70d215849adfc23a
paper_authors.name: Ji Liu
paper_title: BizFinBench: A Business-Driven Real-World Financial Benchmark for
    Evaluating LLMs
paper_summary: Large language models excel in general tasks, yet assessing their reliability
  in logic-heavy, precision-critical domains like finance, law, and healthcare
  remains challenging. To address this, we introduce BizFinBench, the first
  benchmark specifically designed to evaluate LLMs in real-world financial
  applications. BizFinBench consists of 6,781 well-annotated queries in Chinese,
  spanning five dimensions: numerical calculation, reasoning, information
  extraction, prediction recognition, and knowledge-based question answering,
  grouped into nine fine-grained categories. The benchmark includes both
  objective and subjective metrics. We also introduce IteraJudge, a novel LLM
  evaluation method that reduces bias when LLMs serve as evaluators in objective
  metrics. We benchmark 25 models, including both proprietary and open-source
  systems. Extensive experiments show that no model dominates across all tasks.
  Our evaluation reveals distinct capability patterns: (1) In Numerical
  Calculation, Claude-3.5-Sonnet (63.18) and DeepSeek-R1 (64.04) lead, while
  smaller models like Qwen2.5-VL-3B (15.92) lag significantly; (2) In Reasoning,
  proprietary models dominate (ChatGPT-o3: 83.58, Gemini-2.0-Flash: 81.15), with
  open-source models trailing by up to 19.49 points; (3) In Information
  Extraction, the performance spread is the largest, with DeepSeek-R1 scoring
  71.46, while Qwen3-1.7B scores 11.23; (4) In Prediction Recognition,
  performance variance is minimal, with top models scoring between 39.16 and
  50.00. We find that while current LLMs handle routine finance queries
  competently, they struggle with complex scenarios requiring cross-concept
  reasoning. BizFinBench offers a rigorous, business-aligned benchmark for future
  research. The code and dataset are available at
  https://github.com/HiThink-Research/BizFinBench.
paper_ai_summary: BizFinBench is a benchmark for evaluating large language models in financial applications, revealing distinct performance patterns across various tasks.
paper_ai_keywords: large language models BizFinBench numerical calculation reasoning information extraction prediction recognition knowledge-based question answering IteraJudge Claude-3.5-Sonnet DeepSeek-R1 Qwen2.5-VL-3B ChatGPT-o3 Gemini-2.0-Flash Qwen3-1.7B