title: FreshLLMs: Refreshing Large Language Models with Search Engine
    Augmentation
summary: Most large language models (LLMs) are trained once and never updated; thus,
  they lack the ability to dynamically adapt to our ever-changing world. In this
  work, we perform a detailed study of the factuality of LLM-generated text in
  the context of answering questions that test current world knowledge.
  Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a
  diverse range of question and answer types, including questions that require
  fast-changing world knowledge as well as questions with false premises that
  need to be debunked. We benchmark a diverse array of both closed and
  open-source LLMs under a two-mode evaluation procedure that allows us to
  measure both correctness and hallucination. Through human evaluations involving
  more than 50K judgments, we shed light on limitations of these models and
  demonstrate significant room for improvement: for instance, all models
  (regardless of model size) struggle on questions that involve fast-changing
  knowledge and false premises. Motivated by these results, we present
  FreshPrompt, a simple few-shot prompting method that substantially boosts the
  performance of an LLM on FreshQA by incorporating relevant and up-to-date
  information retrieved from a search engine into the prompt. Our experiments
  show that FreshPrompt outperforms both competing search engine-augmented
  prompting methods such as Self-Ask (Press et al., 2022) as well as commercial
  systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that
  both the number of retrieved evidences and their order play a key role in
  influencing the correctness of LLM-generated answers. Additionally, instructing
  the LLM to generate concise and direct answers helps reduce hallucination
  compared to encouraging more verbose answers. To facilitate future work, we
  release FreshQA at github.com/freshllms/freshqa and commit to updating it at
  regular intervals.
paper_authors._id: 651f65f34376ae05d622d76a
paper_authors.name: Tu Vu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:05:19.608Z
paper_authors._id: 636828a644e19ccad213063a
paper_authors.avatarUrl: /avatars/e1551d4ee9ca94018103af3bf1249cf2.svg
paper_authors.fullname: Tu Vu
paper_authors.type: user
paper_authors.user: tuvu
paper_authors._id: 651f65f34376ae05d622d76b
paper_authors.name: Mohit Iyyer
paper_authors._id: 651f65f34376ae05d622d76c
paper_authors.name: Xuezhi Wang
paper_authors._id: 651f65f34376ae05d622d76d
paper_authors.name: Noah Constant
paper_authors._id: 651f65f34376ae05d622d76e
paper_authors.name: Jerry Wei
paper_authors._id: 651f65f34376ae05d622d76f
paper_authors.name: Jason Wei
paper_authors._id: 651f65f34376ae05d622d770
paper_authors.name: Chris Tar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:26:54.499Z
paper_authors._id: 623e5bd5d10f369bb5f2d392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623e5bd5d10f369bb5f2d392/u3nib56InZsTxPrdYPAd1.jpeg
paper_authors.fullname: Chris Tar
paper_authors.type: user
paper_authors.user: ctar
paper_authors._id: 651f65f34376ae05d622d771
paper_authors.name: Yun-Hsuan Sung
paper_authors._id: 651f65f34376ae05d622d772
paper_authors.name: Denny Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T18:27:22.740Z
paper_authors._id: 65d4d62d493aeaf0bb295f1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
paper_authors.fullname: Denny Zhou
paper_authors.type: user
paper_authors.user: dennyzhou
paper_authors._id: 651f65f34376ae05d622d773
paper_authors.name: Quoc Le
paper_authors._id: 651f65f34376ae05d622d774
paper_authors.name: Thang Luong
paper_title: FreshLLMs: Refreshing Large Language Models with Search Engine
    Augmentation
paper_summary: Most large language models (LLMs) are trained once and never updated; thus,
  they lack the ability to dynamically adapt to our ever-changing world. In this
  work, we perform a detailed study of the factuality of LLM-generated text in
  the context of answering questions that test current world knowledge.
  Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a
  diverse range of question and answer types, including questions that require
  fast-changing world knowledge as well as questions with false premises that
  need to be debunked. We benchmark a diverse array of both closed and
  open-source LLMs under a two-mode evaluation procedure that allows us to
  measure both correctness and hallucination. Through human evaluations involving
  more than 50K judgments, we shed light on limitations of these models and
  demonstrate significant room for improvement: for instance, all models
  (regardless of model size) struggle on questions that involve fast-changing
  knowledge and false premises. Motivated by these results, we present
  FreshPrompt, a simple few-shot prompting method that substantially boosts the
  performance of an LLM on FreshQA by incorporating relevant and up-to-date
  information retrieved from a search engine into the prompt. Our experiments
  show that FreshPrompt outperforms both competing search engine-augmented
  prompting methods such as Self-Ask (Press et al., 2022) as well as commercial
  systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that
  both the number of retrieved evidences and their order play a key role in
  influencing the correctness of LLM-generated answers. Additionally, instructing
  the LLM to generate concise and direct answers helps reduce hallucination
  compared to encouraging more verbose answers. To facilitate future work, we
  release FreshQA at github.com/freshllms/freshqa and commit to updating it at
  regular intervals.
paper_ai_summary: FreshQA evaluates LLMs' factuality and hallucination in responding to current knowledge questions, revealing limitations and demonstrating improvements through FreshPrompt, an augmented prompting method using search engine data.
paper_ai_keywords: large language models LLMs dynamic QA benchmark FreshQA fast-changing world knowledge false premises two-mode evaluation human evaluations FreshPrompt Self-Ask Perplexity.AI search engine-augmented prompting