title: UCFE: A User-Centric Financial Expertise Benchmark for Large Language
    Models
summary: This paper introduces the UCFE: User-Centric Financial Expertise benchmark,
  an innovative framework designed to evaluate the ability of large language
  models (LLMs) to handle complex real-world financial tasks. UCFE benchmark
  adopts a hybrid approach that combines human expert evaluations with dynamic,
  task-specific interactions to simulate the complexities of evolving financial
  scenarios. Firstly, we conducted a user study involving 804 participants,
  collecting their feedback on financial tasks. Secondly, based on this feedback,
  we created our dataset that encompasses a wide range of user intents and
  interactions. This dataset serves as the foundation for benchmarking 12 LLM
  services using the LLM-as-Judge methodology. Our results show a significant
  alignment between benchmark scores and human preferences, with a Pearson
  correlation coefficient of 0.78, confirming the effectiveness of the UCFE
  dataset and our evaluation approach. UCFE benchmark not only reveals the
  potential of LLMs in the financial sector but also provides a robust framework
  for assessing their performance and user satisfaction.The benchmark dataset and
  evaluation code are available.
paper_authors._id: 6715fc2cac8cb216a41877a5
paper_authors.name: Yuzhe Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:40:43.520Z
paper_authors._id: 63f622c69cbd6730302783eb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f622c69cbd6730302783eb/9cb96JVKiOm_JhF-shbFw.jpeg
paper_authors.fullname: Yuzhe Yang
paper_authors.type: user
paper_authors.user: TobyYang7
paper_authors._id: 6715fc2cac8cb216a41877a6
paper_authors.name: Yifei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:40:45.362Z
paper_authors._id: 643c047326f177a3e41627b6
paper_authors.avatarUrl: /avatars/ade75cebd049daf080ba80a80d516240.svg
paper_authors.fullname: Yifei Zhang
paper_authors.type: user
paper_authors.user: amstrongzyf
paper_authors._id: 6715fc2cac8cb216a41877a7
paper_authors.name: Yan Hu
paper_authors._id: 6715fc2cac8cb216a41877a8
paper_authors.name: Yilin Guo
paper_authors._id: 6715fc2cac8cb216a41877a9
paper_authors.name: Ruoli Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:04:36.004Z
paper_authors._id: 651fdadbf998876ea7515265
paper_authors.avatarUrl: /avatars/d43c4dc13764a0c43783c3b4e3fcda0e.svg
paper_authors.fullname: ganruoli
paper_authors.type: user
paper_authors.user: wittenberg
paper_authors._id: 6715fc2cac8cb216a41877aa
paper_authors.name: Yueru He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:04:42.030Z
paper_authors._id: 65bd14e8ce846f8aa94db1d1
paper_authors.avatarUrl: /avatars/76eaad15bf32eba75271f3dc315527c2.svg
paper_authors.fullname: Yueru He
paper_authors.type: user
paper_authors.user: Yueru1
paper_authors._id: 6715fc2cac8cb216a41877ab
paper_authors.name: Mingcong Lei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:05:00.431Z
paper_authors._id: 6628c6107751d297d7025a71
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6628c6107751d297d7025a71/S1rm5VIwV2Uxfv8GetKMU.jpeg
paper_authors.fullname: Lei Mingcong
paper_authors.type: user
paper_authors.user: SP4595
paper_authors._id: 6715fc2cac8cb216a41877ac
paper_authors.name: Xiao Zhang
paper_authors._id: 6715fc2cac8cb216a41877ad
paper_authors.name: Haining Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:05:09.237Z
paper_authors._id: 61f40f56b34a114b58db2fed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643384634931-noauth.png
paper_authors.fullname: Haining Wang
paper_authors.type: user
paper_authors.user: haining
paper_authors._id: 6715fc2cac8cb216a41877ae
paper_authors.name: Qianqian Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:05:41.140Z
paper_authors._id: 6479f4317c18dca75e9a9324
paper_authors.avatarUrl: /avatars/9aa709230b057f57ee4415c04a622c63.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: QianqianXie1994
paper_authors._id: 6715fc2cac8cb216a41877af
paper_authors.name: Jimin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:06:03.105Z
paper_authors._id: 63b58ed5889aa6707f0bb0f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b58ed5889aa6707f0bb0f4/znl74_aMswlV8VtHrfj3G.jpeg
paper_authors.fullname: Jimin Huang
paper_authors.type: user
paper_authors.user: jiminHuang
paper_authors._id: 6715fc2cac8cb216a41877b0
paper_authors.name: Honghai Yu
paper_authors._id: 6715fc2cac8cb216a41877b1
paper_authors.name: Benyou Wang
paper_title: UCFE: A User-Centric Financial Expertise Benchmark for Large Language
    Models
paper_summary: This paper introduces the UCFE: User-Centric Financial Expertise benchmark,
  an innovative framework designed to evaluate the ability of large language
  models (LLMs) to handle complex real-world financial tasks. UCFE benchmark
  adopts a hybrid approach that combines human expert evaluations with dynamic,
  task-specific interactions to simulate the complexities of evolving financial
  scenarios. Firstly, we conducted a user study involving 804 participants,
  collecting their feedback on financial tasks. Secondly, based on this feedback,
  we created our dataset that encompasses a wide range of user intents and
  interactions. This dataset serves as the foundation for benchmarking 12 LLM
  services using the LLM-as-Judge methodology. Our results show a significant
  alignment between benchmark scores and human preferences, with a Pearson
  correlation coefficient of 0.78, confirming the effectiveness of the UCFE
  dataset and our evaluation approach. UCFE benchmark not only reveals the
  potential of LLMs in the financial sector but also provides a robust framework
  for assessing their performance and user satisfaction.The benchmark dataset and
  evaluation code are available.