Expect the Unexpected: FailSafe Long Context QA for Finance
We propose a new long-context financial benchmark, FailSafeQA, designed to
test the robustness and context-awareness of LLMs against six variations in
human-interface interactions in LLM-based query-answer systems within finance.
We concentrate on two case studies: Query Failure and Context Failure. In the
Query Failure scenario, we perturb the original query to vary in domain
expertise, completeness, and linguistic accuracy. In the Context Failure case,
we simulate the uploads of degraded, irrelevant, and empty documents. We employ
the LLM-as-a-Judge methodology with Qwen2.5-72B-Instruct and use fine-grained
rating criteria to define and calculate Robustness, Context Grounding, and
Compliance scores for 24 off-the-shelf models. The results suggest that
although some models excel at mitigating input perturbations, they must balance
robust answering with the ability to refrain from hallucinating. Notably,
Palmyra-Fin-128k-Instruct, recognized as the most compliant model, maintained
strong baseline performance but encountered challenges in sustaining robust
predictions in 17% of test cases. On the other hand, the most robust model,
OpenAI o3-mini, fabricated information in 41% of tested cases. The results
demonstrate that even high-performing models have significant room for
improvement and highlight the role of FailSafeQA as a tool for developing LLMs
optimized for dependability in financial applications. The dataset is available
at: https://huggingface.co/datasets/Writer/FailSafeQA
67ab4174757d2eb190af0375
Kiran Kamble
claimed_verified
2025-02-12T09:16:55.367Z
621d6f532165dc431641e438
/avatars/56ccef10a8426d7160ef3586a771bd63.svg
Kiran Kamble
user
kiranr
67ab4174757d2eb190af0376
Melisa Russak
claimed_verified
2025-03-18T08:12:10.481Z
60e61b3969bd0df25c9375da
https://cdn-avatars.huggingface.co/v1/production/uploads/1625692968400-noauth.jpeg
Melisa Russak
user
melisa
67ab4174757d2eb190af0377
Dmytro Mozolevskyi
claimed_verified
2025-02-13T08:25:31.358Z
64f13a7c9be8cab82d9b5a55
/avatars/a00b5d386016697b4c4cc746bac16168.svg
Dmytro Mozolevskyi
user
dmytro-writer
67ab4174757d2eb190af0378
Muayad Ali
claimed_verified
2025-02-12T09:16:53.157Z
6320a906a023aad6a7670e99
/avatars/48071559b0c7660bf6861cfe008b3006.svg
Muayad Sayed Ali
user
muayad
67ab4174757d2eb190af0379
Mateusz Russak
67ab4174757d2eb190af037a
Waseem AlShikh
claimed_verified
2025-02-19T09:05:02.858Z
60cd486d723acf5eb46fe8d3
https://cdn-avatars.huggingface.co/v1/production/uploads/60cd486d723acf5eb46fe8d3/Z1bD1kjvZ0QAOjZna41Xr.jpeg
Waseem AlShikh
user
wassemgtk
Expect the Unexpected: FailSafe Long Context QA for Finance
We propose a new long-context financial benchmark, FailSafeQA, designed to
test the robustness and context-awareness of LLMs against six variations in
human-interface interactions in LLM-based query-answer systems within finance.
We concentrate on two case studies: Query Failure and Context Failure. In the
Query Failure scenario, we perturb the original query to vary in domain
expertise, completeness, and linguistic accuracy. In the Context Failure case,
we simulate the uploads of degraded, irrelevant, and empty documents. We employ
the LLM-as-a-Judge methodology with Qwen2.5-72B-Instruct and use fine-grained
rating criteria to define and calculate Robustness, Context Grounding, and
Compliance scores for 24 off-the-shelf models. The results suggest that
although some models excel at mitigating input perturbations, they must balance
robust answering with the ability to refrain from hallucinating. Notably,
Palmyra-Fin-128k-Instruct, recognized as the most compliant model, maintained
strong baseline performance but encountered challenges in sustaining robust
predictions in 17% of test cases. On the other hand, the most robust model,
OpenAI o3-mini, fabricated information in 41% of tested cases. The results
demonstrate that even high-performing models have significant room for
improvement and highlight the role of FailSafeQA as a tool for developing LLMs
optimized for dependability in financial applications. The dataset is available
at: https://huggingface.co/datasets/Writer/FailSafeQA
FailSafeQA evaluates the robustness and context-awareness of large language models in financial applications through domain expertise, query completeness, linguistic accuracy, and document relevance challenges.
LLM long-context financial benchmark FailSafeQA Query Failure Context Failure LLM-as-a-Judge Qwen2.5-72B-Instruct Robustness Context Grounding Compliance Palmyra-Fin-128k-Instruct OpenAI o3-mini hallucination