Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating
  Financial Large Language Models
As large language models become increasingly prevalent in the financial
sector, there is a pressing need for a standardized method to comprehensively
assess their performance. However, existing finance benchmarks often suffer
from limited language and task coverage, as well as challenges such as
low-quality datasets and inadequate adaptability for LLM evaluation. To address
these limitations, we propose "Golden Touchstone", the first comprehensive
bilingual benchmark for financial LLMs, which incorporates representative
datasets from both Chinese and English across eight core financial NLP tasks.
Developed from extensive open source data collection and industry-specific
demands, this benchmark includes a variety of financial tasks aimed at
thoroughly assessing models' language understanding and generation
capabilities. Through comparative analysis of major models on the benchmark,
such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and
limitations in processing complex financial information. Additionally, we
open-sourced Touchstone-GPT, a financial LLM trained through continual
pre-training and financial instruction tuning, which demonstrates strong
performance on the bilingual benchmark but still has limitations in specific
tasks.This research not only provides the financial large language models with
a practical evaluation tool but also guides the development and optimization of
future research. The source code for Golden Touchstone and model weight of
Touchstone-GPT have been made publicly available at
https://github.com/IDEA-FinAI/Golden-Touchstone, contributing to the
ongoing evolution of FinLLMs and fostering further research in this critical
area.
6732b4e878e78fe8997c13d6
Xiaojun Wu
claimed_verified
2024-11-15T09:27:09.966Z
6352637d0f9bdb641c44e52d
https://cdn-avatars.huggingface.co/v1/production/uploads/6352637d0f9bdb641c44e52d/mSBRPzcH5pIV68PUmcsHV.png
wuxiaojun
user
wuxiaojun
6732b4e878e78fe8997c13d7
Junxi Liu
6732b4e878e78fe8997c13d8
Huanyi Su
6732b4e878e78fe8997c13d9
Zhouchi Lin
6732b4e878e78fe8997c13da
Yiyan Qi
6732b4e878e78fe8997c13db
Chengjin Xu
6732b4e878e78fe8997c13dc
Jiajun Su
6732b4e878e78fe8997c13dd
Jiajie Zhong
6732b4e878e78fe8997c13de
Fuwei Wang
6732b4e878e78fe8997c13df
Saizhuo Wang
6732b4e878e78fe8997c13e0
Fengrui Hua
6732b4e878e78fe8997c13e1
Jia Li
6732b4e878e78fe8997c13e2
Jian Guo
Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating
  Financial Large Language Models
As large language models become increasingly prevalent in the financial
sector, there is a pressing need for a standardized method to comprehensively
assess their performance. However, existing finance benchmarks often suffer
from limited language and task coverage, as well as challenges such as
low-quality datasets and inadequate adaptability for LLM evaluation. To address
these limitations, we propose "Golden Touchstone", the first comprehensive
bilingual benchmark for financial LLMs, which incorporates representative
datasets from both Chinese and English across eight core financial NLP tasks.
Developed from extensive open source data collection and industry-specific
demands, this benchmark includes a variety of financial tasks aimed at
thoroughly assessing models' language understanding and generation
capabilities. Through comparative analysis of major models on the benchmark,
such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and
limitations in processing complex financial information. Additionally, we
open-sourced Touchstone-GPT, a financial LLM trained through continual
pre-training and financial instruction tuning, which demonstrates strong
performance on the bilingual benchmark but still has limitations in specific
tasks.This research not only provides the financial large language models with
a practical evaluation tool but also guides the development and optimization of
future research. The source code for Golden Touchstone and model weight of
Touchstone-GPT have been made publicly available at
https://github.com/IDEA-FinAI/Golden-Touchstone, contributing to the
ongoing evolution of FinLLMs and fostering further research in this critical
area.
Golden Touchstone is a comprehensive bilingual benchmark for financial LLMs, addressing limitations of existing benchmarks and providing a practical evaluation tool for models' performance across multiple tasks.
financial LLMs Golden Touchstone bilingual benchmark financial NLP tasks GPT-4 Llama3 FinGPT FinMA continual pre-training financial instruction tuning Touchstone-GPT