title: SealQA: Raising the Bar for Reasoning in Search-Augmented Language
    Models
summary: We introduce SealQA, a new challenge benchmark for evaluating
  SEarch-Augmented Language models on fact-seeking questions where web search
  yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors:
  (1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and
  reasoning capabilities, with Seal-0 focusing on the most challenging questions
  where chat models (e.g., GPT-4.1) typically achieve near-zero accuracy; and (3)
  LongSeal, which extends SealQA to test long-context, multi-document reasoning
  in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations
  in current models: Even frontier LLMs perform poorly across all SealQA flavors.
  On Seal-0, frontier agentic models equipped with tools like o3 and o4-mini
  achieve only 17.1% and 6.3% accuracy, respectively, at their best reasoning
  efforts. We find that advanced reasoning models such as DeepSeek-R1-671B and
  o3-mini are highly vulnerable to noisy search results. Notably, increasing
  test-time compute does not yield reliable gains across o3-mini, o4-mini, and
  o3, with performance often plateauing or even declining early. Additionally,
  while recent models are less affected by the "lost-in-the-middle" issue, they
  still fail to reliably identify relevant documents in LongSeal when faced with
  numerous distractors. To facilitate future work, we release SealQA at
  huggingface.co/datasets/vtllms/sealqa.
paper_authors._id: 683f3b76b5399dccb436f5e9
paper_authors.name: Thinh Pham
paper_authors._id: 683f3b76b5399dccb436f5ea
paper_authors.name: Nguyen Nguyen
paper_authors._id: 683f3b76b5399dccb436f5eb
paper_authors.name: Pratibha Zunjare
paper_authors._id: 683f3b76b5399dccb436f5ec
paper_authors.name: Weiyuan Chen
paper_authors._id: 683f3b76b5399dccb436f5ed
paper_authors.name: Yu-Min Tseng
paper_authors._id: 683f3b76b5399dccb436f5ee
paper_authors.name: Tu Vu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:01:23.482Z
paper_authors._id: 636828a644e19ccad213063a
paper_authors.avatarUrl: /avatars/e1551d4ee9ca94018103af3bf1249cf2.svg
paper_authors.fullname: Tu Vu
paper_authors.type: user
paper_authors.user: tuvu
paper_title: SealQA: Raising the Bar for Reasoning in Search-Augmented Language
    Models
paper_summary: We introduce SealQA, a new challenge benchmark for evaluating
  SEarch-Augmented Language models on fact-seeking questions where web search
  yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors:
  (1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and
  reasoning capabilities, with Seal-0 focusing on the most challenging questions
  where chat models (e.g., GPT-4.1) typically achieve near-zero accuracy; and (3)
  LongSeal, which extends SealQA to test long-context, multi-document reasoning
  in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations
  in current models: Even frontier LLMs perform poorly across all SealQA flavors.
  On Seal-0, frontier agentic models equipped with tools like o3 and o4-mini
  achieve only 17.1% and 6.3% accuracy, respectively, at their best reasoning
  efforts. We find that advanced reasoning models such as DeepSeek-R1-671B and
  o3-mini are highly vulnerable to noisy search results. Notably, increasing
  test-time compute does not yield reliable gains across o3-mini, o4-mini, and
  o3, with performance often plateauing or even declining early. Additionally,
  while recent models are less affected by the "lost-in-the-middle" issue, they
  still fail to reliably identify relevant documents in LongSeal when faced with
  numerous distractors. To facilitate future work, we release SealQA at
  huggingface.co/datasets/vtllms/sealqa.
paper_ai_summary: SealQA evaluates search-augmented language models' performance on fact-seeking questions with conflicting or noisy search results, revealing limitations in reasoning and factual accuracy.
paper_ai_keywords: SEarch-Augmented Language models fact-seeking questions SealQA Seal-0 Seal-Hard LongSeal factual accuracy reasoning capabilities chat models GPT-4.1 tools o3 o4-mini DeepSeek-R1-671B test-time compute relevant documents huggingface.co/datasets/vtllms/sealqa