title: OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in
    Financial Domain
summary: As a typical and practical application of Large Language Models (LLMs),
  Retrieval-Augmented Generation (RAG) techniques have gained extensive
  attention, particularly in vertical domains where LLMs may lack domain-specific
  knowledge. In this paper, we introduce an omnidirectional and automatic RAG
  benchmark, OmniEval, in the financial domain. Our benchmark is characterized by
  its multi-dimensional evaluation framework, including (1) a matrix-based RAG
  scenario evaluation system that categorizes queries into five task classes and
  16 financial topics, leading to a structured assessment of diverse query
  scenarios; (2) a multi-dimensional evaluation data generation approach, which
  combines GPT-4-based automatic generation and human annotation, achieving an
  87.47\% acceptance ratio in human evaluations on generated instances; (3) a
  multi-stage evaluation system that evaluates both retrieval and generation
  performance, result in a comprehensive evaluation on the RAG pipeline; and (4)
  robust evaluation metrics derived from rule-based and LLM-based ones, enhancing
  the reliability of assessments through manual annotations and supervised
  fine-tuning of an LLM evaluator. Our experiments demonstrate the
  comprehensiveness of OmniEval, which includes extensive test datasets and
  highlights the performance variations of RAG systems across diverse topics and
  tasks, revealing significant opportunities for RAG models to improve their
  capabilities in vertical domains. We open source the code of our benchmark in
  https://github.com/RUC-NLPIR/OmniEval{https://github.com/RUC-NLPIR/OmniEval}.
paper_authors._id: 67625f697fa2dfcfa4a26eb1
paper_authors.name: Shuting Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:51:53.983Z
paper_authors._id: 66d81fe818e470c7312c0f2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66d81fe818e470c7312c0f2a/HwPcM_yDB0nAgjz9bT_Cs.jpeg
paper_authors.fullname: ShutingWang
paper_authors.type: user
paper_authors.user: ShootingWong
paper_authors._id: 67625f697fa2dfcfa4a26eb2
paper_authors.name: Jiejun Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T12:38:31.592Z
paper_authors._id: 62e52483a944e2a56cd2c6ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e52483a944e2a56cd2c6ca/pG44O-1qD00q5CEJMMyFQ.jpeg
paper_authors.fullname: Jiejun Tan
paper_authors.type: user
paper_authors.user: zstanjj
paper_authors._id: 67625f697fa2dfcfa4a26eb3
paper_authors.name: Zhicheng Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:52:00.291Z
paper_authors._id: 66f0bf59e9d50ec57febf751
paper_authors.avatarUrl: /avatars/be97941e60064e5dd806c6fe9db3c537.svg
paper_authors.fullname: Zhicheng Dou
paper_authors.type: user
paper_authors.user: douzc
paper_authors._id: 67625f697fa2dfcfa4a26eb4
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:52:06.433Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_title: OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in
    Financial Domain
paper_summary: As a typical and practical application of Large Language Models (LLMs),
  Retrieval-Augmented Generation (RAG) techniques have gained extensive
  attention, particularly in vertical domains where LLMs may lack domain-specific
  knowledge. In this paper, we introduce an omnidirectional and automatic RAG
  benchmark, OmniEval, in the financial domain. Our benchmark is characterized by
  its multi-dimensional evaluation framework, including (1) a matrix-based RAG
  scenario evaluation system that categorizes queries into five task classes and
  16 financial topics, leading to a structured assessment of diverse query
  scenarios; (2) a multi-dimensional evaluation data generation approach, which
  combines GPT-4-based automatic generation and human annotation, achieving an
  87.47\% acceptance ratio in human evaluations on generated instances; (3) a
  multi-stage evaluation system that evaluates both retrieval and generation
  performance, result in a comprehensive evaluation on the RAG pipeline; and (4)
  robust evaluation metrics derived from rule-based and LLM-based ones, enhancing
  the reliability of assessments through manual annotations and supervised
  fine-tuning of an LLM evaluator. Our experiments demonstrate the
  comprehensiveness of OmniEval, which includes extensive test datasets and
  highlights the performance variations of RAG systems across diverse topics and
  tasks, revealing significant opportunities for RAG models to improve their
  capabilities in vertical domains. We open source the code of our benchmark in
  https://github.com/RUC-NLPIR/OmniEval{https://github.com/RUC-NLPIR/OmniEval}.
paper_ai_summary: OmniEval is a comprehensive benchmark for Retrieval-Augmented Generation (RAG) in the financial domain, offering a structured evaluation framework with automatic generation, human annotation, and robust metrics.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) OmniEval GPT-4 multi-dimensional evaluation scenario evaluation financial topics multi-stage evaluation evaluation metrics LLM evaluator supervised fine-tuning