title: Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating
    Financial Large Language Models
summary: As large language models become increasingly prevalent in the financial
  sector, there is a pressing need for a standardized method to comprehensively
  assess their performance. However, existing finance benchmarks often suffer
  from limited language and task coverage, as well as challenges such as
  low-quality datasets and inadequate adaptability for LLM evaluation. To address
  these limitations, we propose "Golden Touchstone", the first comprehensive
  bilingual benchmark for financial LLMs, which incorporates representative
  datasets from both Chinese and English across eight core financial NLP tasks.
  Developed from extensive open source data collection and industry-specific
  demands, this benchmark includes a variety of financial tasks aimed at
  thoroughly assessing models' language understanding and generation
  capabilities. Through comparative analysis of major models on the benchmark,
  such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and
  limitations in processing complex financial information. Additionally, we
  open-sourced Touchstone-GPT, a financial LLM trained through continual
  pre-training and financial instruction tuning, which demonstrates strong
  performance on the bilingual benchmark but still has limitations in specific
  tasks.This research not only provides the financial large language models with
  a practical evaluation tool but also guides the development and optimization of
  future research. The source code for Golden Touchstone and model weight of
  Touchstone-GPT have been made publicly available at
  https://github.com/IDEA-FinAI/Golden-Touchstone, contributing to the
  ongoing evolution of FinLLMs and fostering further research in this critical
  area.
paper_authors._id: 6732b4e878e78fe8997c13d6
paper_authors.name: Xiaojun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:27:09.966Z
paper_authors._id: 6352637d0f9bdb641c44e52d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6352637d0f9bdb641c44e52d/mSBRPzcH5pIV68PUmcsHV.png
paper_authors.fullname: wuxiaojun
paper_authors.type: user
paper_authors.user: wuxiaojun
paper_authors._id: 6732b4e878e78fe8997c13d7
paper_authors.name: Junxi Liu
paper_authors._id: 6732b4e878e78fe8997c13d8
paper_authors.name: Huanyi Su
paper_authors._id: 6732b4e878e78fe8997c13d9
paper_authors.name: Zhouchi Lin
paper_authors._id: 6732b4e878e78fe8997c13da
paper_authors.name: Yiyan Qi
paper_authors._id: 6732b4e878e78fe8997c13db
paper_authors.name: Chengjin Xu
paper_authors._id: 6732b4e878e78fe8997c13dc
paper_authors.name: Jiajun Su
paper_authors._id: 6732b4e878e78fe8997c13dd
paper_authors.name: Jiajie Zhong
paper_authors._id: 6732b4e878e78fe8997c13de
paper_authors.name: Fuwei Wang
paper_authors._id: 6732b4e878e78fe8997c13df
paper_authors.name: Saizhuo Wang
paper_authors._id: 6732b4e878e78fe8997c13e0
paper_authors.name: Fengrui Hua
paper_authors._id: 6732b4e878e78fe8997c13e1
paper_authors.name: Jia Li
paper_authors._id: 6732b4e878e78fe8997c13e2
paper_authors.name: Jian Guo
paper_title: Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating
    Financial Large Language Models
paper_summary: As large language models become increasingly prevalent in the financial
  sector, there is a pressing need for a standardized method to comprehensively
  assess their performance. However, existing finance benchmarks often suffer
  from limited language and task coverage, as well as challenges such as
  low-quality datasets and inadequate adaptability for LLM evaluation. To address
  these limitations, we propose "Golden Touchstone", the first comprehensive
  bilingual benchmark for financial LLMs, which incorporates representative
  datasets from both Chinese and English across eight core financial NLP tasks.
  Developed from extensive open source data collection and industry-specific
  demands, this benchmark includes a variety of financial tasks aimed at
  thoroughly assessing models' language understanding and generation
  capabilities. Through comparative analysis of major models on the benchmark,
  such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and
  limitations in processing complex financial information. Additionally, we
  open-sourced Touchstone-GPT, a financial LLM trained through continual
  pre-training and financial instruction tuning, which demonstrates strong
  performance on the bilingual benchmark but still has limitations in specific
  tasks.This research not only provides the financial large language models with
  a practical evaluation tool but also guides the development and optimization of
  future research. The source code for Golden Touchstone and model weight of
  Touchstone-GPT have been made publicly available at
  https://github.com/IDEA-FinAI/Golden-Touchstone, contributing to the
  ongoing evolution of FinLLMs and fostering further research in this critical
  area.
paper_ai_summary: Golden Touchstone is a comprehensive bilingual benchmark for financial LLMs, addressing limitations of existing benchmarks and providing a practical evaluation tool for models' performance across multiple tasks.
paper_ai_keywords: financial LLMs Golden Touchstone bilingual benchmark financial NLP tasks GPT-4 Llama3 FinGPT FinMA continual pre-training financial instruction tuning Touchstone-GPT