title: The FinBen: An Holistic Financial Benchmark for Large Language Models
summary: LLMs have transformed NLP and shown promise in various fields, yet their
  potential in finance is underexplored due to a lack of thorough evaluations and
  the complexity of financial tasks. This along with the rapid development of
  LLMs, highlights the urgent need for a systematic financial evaluation
  benchmark for LLMs. In this paper, we introduce FinBen, the first comprehensive
  open-sourced evaluation benchmark, specifically designed to thoroughly assess
  the capabilities of LLMs in the financial domain. FinBen encompasses 35
  datasets across 23 financial tasks, organized into three spectrums of
  difficulty inspired by the Cattell-Horn-Carroll theory, to evaluate LLMs'
  cognitive abilities in inductive reasoning, associative memory, quantitative
  reasoning, crystallized intelligence, and more. Our evaluation of 15
  representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals
  insights into their strengths and limitations within the financial domain. The
  findings indicate that GPT-4 leads in quantification, extraction, numerical
  reasoning, and stock trading, while Gemini shines in generation and
  forecasting; however, both struggle with complex extraction and forecasting,
  showing a clear need for targeted enhancements. Instruction tuning boosts
  simple task performance but falls short in improving complex reasoning and
  forecasting abilities. FinBen seeks to continuously evaluate LLMs in finance,
  fostering AI development with regular updates of tasks and models.
paper_authors._id: 65d56a82acacbc30a2eb627c
paper_authors.name: Qianqian Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-25T09:44:31.521Z
paper_authors._id: 6479f4317c18dca75e9a9324
paper_authors.avatarUrl: /avatars/9aa709230b057f57ee4415c04a622c63.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: QianqianXie1994
paper_authors._id: 65d56a82acacbc30a2eb627d
paper_authors.name: Weiguang Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:03:28.892Z
paper_authors._id: 6353a8e7eca5a823e3b578b6
paper_authors.avatarUrl: /avatars/c533748aad297db0c0273c3b78f42ab4.svg
paper_authors.fullname: Weiguang Han
paper_authors.type: user
paper_authors.user: tothemoon96
paper_authors._id: 65d56a82acacbc30a2eb627e
paper_authors.name: Zhengyu Chen
paper_authors._id: 65d56a82acacbc30a2eb627f
paper_authors.name: Ruoyu Xiang
paper_authors._id: 65d56a82acacbc30a2eb6280
paper_authors.name: Xiao Zhang
paper_authors._id: 65d56a82acacbc30a2eb6281
paper_authors.name: Yueru He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:04:49.547Z
paper_authors._id: 65bd14e8ce846f8aa94db1d1
paper_authors.avatarUrl: /avatars/76eaad15bf32eba75271f3dc315527c2.svg
paper_authors.fullname: Yueru He
paper_authors.type: user
paper_authors.user: Yueru1
paper_authors._id: 65d56a82acacbc30a2eb6282
paper_authors.name: Mengxi Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:38:07.273Z
paper_authors._id: 663adb42e14047f710dc1d29
paper_authors.avatarUrl: /avatars/7ca49d67a4a8b4cf0ee896e07646715f.svg
paper_authors.fullname: Mengxi Xiao
paper_authors.type: user
paper_authors.user: ElsaShaw
paper_authors._id: 65d56a82acacbc30a2eb6283
paper_authors.name: Dong Li
paper_authors._id: 65d56a82acacbc30a2eb6284
paper_authors.name: Yongfu Dai
paper_authors._id: 65d56a82acacbc30a2eb6285
paper_authors.name: Duanyu Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:05:57.431Z
paper_authors._id: 649d96407b4d8f568b49f472
paper_authors.avatarUrl: /avatars/c0ca58462749108b5db9538b4be6d9ac.svg
paper_authors.fullname: Duanyu Feng
paper_authors.type: user
paper_authors.user: ColFeng
paper_authors._id: 65d56a82acacbc30a2eb6286
paper_authors.name: Yijing Xu
paper_authors._id: 65d56a82acacbc30a2eb6287
paper_authors.name: Haoqiang Kang
paper_authors._id: 65d56a82acacbc30a2eb6288
paper_authors.name: Ziyan Kuang
paper_authors._id: 65d56a82acacbc30a2eb6289
paper_authors.name: Chenhan Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:06:57.108Z
paper_authors._id: 6447fbe54988ee01f2a59381
paper_authors.avatarUrl: /avatars/bbf497b37b1112da111192caf4ebbe1a.svg
paper_authors.fullname: Chenhan Yuan
paper_authors.type: user
paper_authors.user: chrisyuan45
paper_authors._id: 65d56a82acacbc30a2eb628a
paper_authors.name: Kailai Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:07:04.217Z
paper_authors._id: 646fc402e9c03ba436d5e93e
paper_authors.avatarUrl: /avatars/870c86dc99fb1cb6a348a7a0385b1a04.svg
paper_authors.fullname: Kailai Yang
paper_authors.type: user
paper_authors.user: klyang
paper_authors._id: 65d56a82acacbc30a2eb628b
paper_authors.name: Zheheng Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:07:33.064Z
paper_authors._id: 6443bb593c323e0918f61a96
paper_authors.avatarUrl: /avatars/b9e1ba17f7798b5142bc0124fba95237.svg
paper_authors.fullname: zheheng luo
paper_authors.type: user
paper_authors.user: KenLuo
paper_authors._id: 65d56a82acacbc30a2eb628c
paper_authors.name: Tianlin Zhang
paper_authors._id: 65d56a82acacbc30a2eb628d
paper_authors.name: Zhiwei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-04T09:00:41.436Z
paper_authors._id: 654d10d918c4ae0b4017fa4a
paper_authors.avatarUrl: /avatars/a905ef9419ecffab40b17bdfcf23dce2.svg
paper_authors.fullname: Zhiwei Liu
paper_authors.type: user
paper_authors.user: lzw1008
paper_authors._id: 65d56a82acacbc30a2eb628e
paper_authors.name: Guojun Xiong
paper_authors._id: 65d56a82acacbc30a2eb628f
paper_authors.name: Zhiyang Deng
paper_authors._id: 65d56a82acacbc30a2eb6290
paper_authors.name: Yuechen Jiang
paper_authors._id: 65d56a82acacbc30a2eb6291
paper_authors.name: Zhiyuan Yao
paper_authors._id: 65d56a82acacbc30a2eb6292
paper_authors.name: Haohang Li
paper_authors._id: 65d56a82acacbc30a2eb6293
paper_authors.name: Yangyang Yu
paper_authors._id: 65d56a82acacbc30a2eb6294
paper_authors.name: Gang Hu
paper_authors._id: 65d56a82acacbc30a2eb6295
paper_authors.name: Jiajia Huang
paper_authors._id: 65d56a82acacbc30a2eb6296
paper_authors.name: Xiao-Yang Liu
paper_authors._id: 65d56a82acacbc30a2eb6297
paper_authors.name: Alejandro Lopez-Lira
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:10:25.133Z
paper_authors._id: 65b473e5e602b6c2c92cbc8e
paper_authors.avatarUrl: /avatars/a80556ea72f19289be49321f44f0c43a.svg
paper_authors.fullname: Alejandro Lopez-Lira
paper_authors.type: user
paper_authors.user: alejandroll10
paper_authors._id: 65d56a82acacbc30a2eb6298
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:11:09.250Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 65d56a82acacbc30a2eb6299
paper_authors.name: Yanzhao Lai
paper_authors._id: 65d56a82acacbc30a2eb629a
paper_authors.name: Hao Wang
paper_authors._id: 65d56a82acacbc30a2eb629b
paper_authors.name: Min Peng
paper_authors._id: 65d56a82acacbc30a2eb629c
paper_authors.name: Sophia Ananiadou
paper_authors._id: 65d56a82acacbc30a2eb629d
paper_authors.name: Jimin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T12:11:48.539Z
paper_authors._id: 63b58ed5889aa6707f0bb0f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b58ed5889aa6707f0bb0f4/znl74_aMswlV8VtHrfj3G.jpeg
paper_authors.fullname: Jimin Huang
paper_authors.type: user
paper_authors.user: jiminHuang
paper_title: The FinBen: An Holistic Financial Benchmark for Large Language Models
paper_summary: LLMs have transformed NLP and shown promise in various fields, yet their
  potential in finance is underexplored due to a lack of thorough evaluations and
  the complexity of financial tasks. This along with the rapid development of
  LLMs, highlights the urgent need for a systematic financial evaluation
  benchmark for LLMs. In this paper, we introduce FinBen, the first comprehensive
  open-sourced evaluation benchmark, specifically designed to thoroughly assess
  the capabilities of LLMs in the financial domain. FinBen encompasses 35
  datasets across 23 financial tasks, organized into three spectrums of
  difficulty inspired by the Cattell-Horn-Carroll theory, to evaluate LLMs'
  cognitive abilities in inductive reasoning, associative memory, quantitative
  reasoning, crystallized intelligence, and more. Our evaluation of 15
  representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals
  insights into their strengths and limitations within the financial domain. The
  findings indicate that GPT-4 leads in quantification, extraction, numerical
  reasoning, and stock trading, while Gemini shines in generation and
  forecasting; however, both struggle with complex extraction and forecasting,
  showing a clear need for targeted enhancements. Instruction tuning boosts
  simple task performance but falls short in improving complex reasoning and
  forecasting abilities. FinBen seeks to continuously evaluate LLMs in finance,
  fostering AI development with regular updates of tasks and models.
paper_ai_keywords: FinBen Cattell-Horn-Carroll theory inductive reasoning associative memory quantitative reasoning crystallized intelligence GPT-4 ChatGPT Gemini quantification extraction numerical reasoning stock trading generation forecasting instruction tuning