title: Search Arena: Analyzing Search-Augmented LLMs
summary: Search-augmented language models combine web search with Large Language
  Models (LLMs) to improve response groundedness and freshness. However,
  analyzing these systems remains challenging: existing datasets are limited in
  scale and narrow in scope, often constrained to static, single-turn,
  fact-checking questions. In this work, we introduce Search Arena, a
  crowd-sourced, large-scale, human-preference dataset of over 24,000 paired
  multi-turn user interactions with search-augmented LLMs. The dataset spans
  diverse intents and languages, and contains full system traces with around
  12,000 human preference votes. Our analysis reveals that user preferences are
  influenced by the number of citations, even when the cited content does not
  directly support the attributed claims, uncovering a gap between perceived and
  actual credibility. Furthermore, user preferences vary across cited sources,
  revealing that community-driven platforms are generally preferred and static
  encyclopedic sources are not always appropriate and reliable. To assess
  performance across different settings, we conduct cross-arena analyses by
  testing search-augmented LLMs in a general-purpose chat environment and
  conventional LLMs in search-intensive settings. We find that web search does
  not degrade and may even improve performance in non-search settings; however,
  the quality in search settings is significantly affected if solely relying on
  the model's parametric knowledge. We open-sourced the dataset to support future
  research in this direction. Our dataset and code are available at:
  https://github.com/lmarena/search-arena.
paper_authors._id: 68430c258f9ec8394c514870
paper_authors.name: Mihran Miroyan
paper_authors._id: 68430c258f9ec8394c514871
paper_authors.name: Tsung-Han Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:45:14.475Z
paper_authors._id: 644a767044b75fd95805d232
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644a767044b75fd95805d232/vHA2vI_B3CpXapdBEwspB.jpeg
paper_authors.fullname: Patrick (Tsung-Han) Wu
paper_authors.type: user
paper_authors.user: tsunghanwu
paper_authors._id: 68430c258f9ec8394c514872
paper_authors.name: Logan King
paper_authors._id: 68430c258f9ec8394c514873
paper_authors.name: Tianle Li
paper_authors._id: 68430c258f9ec8394c514874
paper_authors.name: Jiayi Pan
paper_authors._id: 68430c258f9ec8394c514875
paper_authors.name: Xinyan Hu
paper_authors._id: 68430c258f9ec8394c514876
paper_authors.name: Wei-Lin Chiang
paper_authors._id: 68430c258f9ec8394c514877
paper_authors.name: Anastasios N. Angelopoulos
paper_authors._id: 68430c258f9ec8394c514878
paper_authors.name: Trevor Darrell
paper_authors._id: 68430c258f9ec8394c514879
paper_authors.name: Narges Norouzi
paper_authors._id: 68430c258f9ec8394c51487a
paper_authors.name: Joseph E. Gonzalez
paper_title: Search Arena: Analyzing Search-Augmented LLMs
paper_summary: Search-augmented language models combine web search with Large Language
  Models (LLMs) to improve response groundedness and freshness. However,
  analyzing these systems remains challenging: existing datasets are limited in
  scale and narrow in scope, often constrained to static, single-turn,
  fact-checking questions. In this work, we introduce Search Arena, a
  crowd-sourced, large-scale, human-preference dataset of over 24,000 paired
  multi-turn user interactions with search-augmented LLMs. The dataset spans
  diverse intents and languages, and contains full system traces with around
  12,000 human preference votes. Our analysis reveals that user preferences are
  influenced by the number of citations, even when the cited content does not
  directly support the attributed claims, uncovering a gap between perceived and
  actual credibility. Furthermore, user preferences vary across cited sources,
  revealing that community-driven platforms are generally preferred and static
  encyclopedic sources are not always appropriate and reliable. To assess
  performance across different settings, we conduct cross-arena analyses by
  testing search-augmented LLMs in a general-purpose chat environment and
  conventional LLMs in search-intensive settings. We find that web search does
  not degrade and may even improve performance in non-search settings; however,
  the quality in search settings is significantly affected if solely relying on
  the model's parametric knowledge. We open-sourced the dataset to support future
  research in this direction. Our dataset and code are available at:
  https://github.com/lmarena/search-arena.
paper_ai_summary: Search Arena is a large-scale human-preference dataset that analyzes user interactions with search-augmented language models, revealing insights into citation influence and source credibility.
paper_ai_keywords: LLMs search-augmented language models dataset human-preference user interactions citations credibility community-driven platforms search-intensive settings parametric knowledge