title: Expect the Unexpected: FailSafe Long Context QA for Finance
summary: We propose a new long-context financial benchmark, FailSafeQA, designed to
  test the robustness and context-awareness of LLMs against six variations in
  human-interface interactions in LLM-based query-answer systems within finance.
  We concentrate on two case studies: Query Failure and Context Failure. In the
  Query Failure scenario, we perturb the original query to vary in domain
  expertise, completeness, and linguistic accuracy. In the Context Failure case,
  we simulate the uploads of degraded, irrelevant, and empty documents. We employ
  the LLM-as-a-Judge methodology with Qwen2.5-72B-Instruct and use fine-grained
  rating criteria to define and calculate Robustness, Context Grounding, and
  Compliance scores for 24 off-the-shelf models. The results suggest that
  although some models excel at mitigating input perturbations, they must balance
  robust answering with the ability to refrain from hallucinating. Notably,
  Palmyra-Fin-128k-Instruct, recognized as the most compliant model, maintained
  strong baseline performance but encountered challenges in sustaining robust
  predictions in 17% of test cases. On the other hand, the most robust model,
  OpenAI o3-mini, fabricated information in 41% of tested cases. The results
  demonstrate that even high-performing models have significant room for
  improvement and highlight the role of FailSafeQA as a tool for developing LLMs
  optimized for dependability in financial applications. The dataset is available
  at: https://huggingface.co/datasets/Writer/FailSafeQA
paper_authors._id: 67ab4174757d2eb190af0375
paper_authors.name: Kiran Kamble
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-12T09:16:55.367Z
paper_authors._id: 621d6f532165dc431641e438
paper_authors.avatarUrl: /avatars/56ccef10a8426d7160ef3586a771bd63.svg
paper_authors.fullname: Kiran Kamble
paper_authors.type: user
paper_authors.user: kiranr
paper_authors._id: 67ab4174757d2eb190af0376
paper_authors.name: Melisa Russak
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:12:10.481Z
paper_authors._id: 60e61b3969bd0df25c9375da
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625692968400-noauth.jpeg
paper_authors.fullname: Melisa Russak
paper_authors.type: user
paper_authors.user: melisa
paper_authors._id: 67ab4174757d2eb190af0377
paper_authors.name: Dmytro Mozolevskyi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:25:31.358Z
paper_authors._id: 64f13a7c9be8cab82d9b5a55
paper_authors.avatarUrl: /avatars/a00b5d386016697b4c4cc746bac16168.svg
paper_authors.fullname: Dmytro Mozolevskyi
paper_authors.type: user
paper_authors.user: dmytro-writer
paper_authors._id: 67ab4174757d2eb190af0378
paper_authors.name: Muayad Ali
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-12T09:16:53.157Z
paper_authors._id: 6320a906a023aad6a7670e99
paper_authors.avatarUrl: /avatars/48071559b0c7660bf6861cfe008b3006.svg
paper_authors.fullname: Muayad Sayed Ali
paper_authors.type: user
paper_authors.user: muayad
paper_authors._id: 67ab4174757d2eb190af0379
paper_authors.name: Mateusz Russak
paper_authors._id: 67ab4174757d2eb190af037a
paper_authors.name: Waseem AlShikh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:05:02.858Z
paper_authors._id: 60cd486d723acf5eb46fe8d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60cd486d723acf5eb46fe8d3/Z1bD1kjvZ0QAOjZna41Xr.jpeg
paper_authors.fullname: Waseem AlShikh
paper_authors.type: user
paper_authors.user: wassemgtk
paper_title: Expect the Unexpected: FailSafe Long Context QA for Finance
paper_summary: We propose a new long-context financial benchmark, FailSafeQA, designed to
  test the robustness and context-awareness of LLMs against six variations in
  human-interface interactions in LLM-based query-answer systems within finance.
  We concentrate on two case studies: Query Failure and Context Failure. In the
  Query Failure scenario, we perturb the original query to vary in domain
  expertise, completeness, and linguistic accuracy. In the Context Failure case,
  we simulate the uploads of degraded, irrelevant, and empty documents. We employ
  the LLM-as-a-Judge methodology with Qwen2.5-72B-Instruct and use fine-grained
  rating criteria to define and calculate Robustness, Context Grounding, and
  Compliance scores for 24 off-the-shelf models. The results suggest that
  although some models excel at mitigating input perturbations, they must balance
  robust answering with the ability to refrain from hallucinating. Notably,
  Palmyra-Fin-128k-Instruct, recognized as the most compliant model, maintained
  strong baseline performance but encountered challenges in sustaining robust
  predictions in 17% of test cases. On the other hand, the most robust model,
  OpenAI o3-mini, fabricated information in 41% of tested cases. The results
  demonstrate that even high-performing models have significant room for
  improvement and highlight the role of FailSafeQA as a tool for developing LLMs
  optimized for dependability in financial applications. The dataset is available
  at: https://huggingface.co/datasets/Writer/FailSafeQA
paper_ai_keywords: LLMs FailSafeQA Query Failure Context Failure domain expertise linguistic accuracy degraded documents irrelevant documents empty documents LLM-as-a-Judge Qwen2.5-72B-Instruct fine-grained rating criteria Robustness Context Grounding Compliance Palmyra-Fin-128k-Instruct hallucinating OpenAI o3-mini