title: Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health
    Information
summary: As Large Language Models (LLMs) become widely accessible, a detailed
  understanding of their knowledge within specific domains becomes necessary for
  successful real world use. This is particularly critical in public health,
  where failure to retrieve relevant, accurate, and current information could
  significantly impact UK residents. However, currently little is known about LLM
  knowledge of UK Government public health information. To address this issue,
  this paper introduces a new benchmark, PubHealthBench, with over 8000 questions
  for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form
  responses to public health queries, created via an automated pipeline. We also
  release a new dataset of the extracted UK Government public health guidance
  documents used as source text for PubHealthBench. Assessing 24 LLMs on
  PubHealthBench we find the latest private LLMs (GPT-4.5, GPT-4.1 and o1) have a
  high degree of knowledge, achieving >90% in the MCQA setup, and outperform
  humans with cursory search engine use. However, in the free form setup we see
  lower performance with no model scoring >75%. Therefore, whilst there are
  promising signs that state of the art (SOTA) LLMs are an increasingly accurate
  source of public health information, additional safeguards or tools may still
  be needed when providing free form responses on public health topics.
paper_authors._id: 6821af48696b63e207ae8474
paper_authors.name: Joshua Harris
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T09:03:48.631Z
paper_authors._id: 64cb98c6f103036e23c69b1d
paper_authors.avatarUrl: /avatars/7ee33880ad39f5335b618dc53554124a.svg
paper_authors.fullname: Harris
paper_authors.type: user
paper_authors.user: Joshua-Harris
paper_authors._id: 6821af48696b63e207ae8475
paper_authors.name: Fan Grayson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-12T13:17:30.555Z
paper_authors._id: 6422f276320bd601bff05691
paper_authors.avatarUrl: /avatars/7f43ca6cb485a73e3de8a0653f109735.svg
paper_authors.fullname: Fan Grayson
paper_authors.type: user
paper_authors.user: fangrayson
paper_authors._id: 6821af48696b63e207ae8476
paper_authors.name: Felix Feldman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-12T13:17:36.880Z
paper_authors._id: 681205b04119e4ecc947632d
paper_authors.avatarUrl: /avatars/94f18aeed6fef7764252e06244e1e7c9.svg
paper_authors.fullname: Felix Feldman
paper_authors.type: user
paper_authors.user: felixFeldman
paper_authors._id: 6821af48696b63e207ae8477
paper_authors.name: Timothy Laurence
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-12T13:17:44.371Z
paper_authors._id: 64b9323691876a8d16d2dae5
paper_authors.avatarUrl: /avatars/ea583af4ec8e5ad18fefbec2281ae50e.svg
paper_authors.fullname: Timothy Laurence
paper_authors.type: user
paper_authors.user: TRLaurence
paper_authors._id: 6821af48696b63e207ae8478
paper_authors.name: Toby Nonnenmacher
paper_authors._id: 6821af48696b63e207ae8479
paper_authors.name: Oliver Higgins
paper_authors._id: 6821af48696b63e207ae847a
paper_authors.name: Leo Loman
paper_authors._id: 6821af48696b63e207ae847b
paper_authors.name: Selina Patel
paper_authors._id: 6821af48696b63e207ae847c
paper_authors.name: Thomas Finnie
paper_authors._id: 6821af48696b63e207ae847d
paper_authors.name: Samuel Collins
paper_authors._id: 6821af48696b63e207ae847e
paper_authors.name: Michael Borowitz
paper_title: Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health
    Information
paper_summary: As Large Language Models (LLMs) become widely accessible, a detailed
  understanding of their knowledge within specific domains becomes necessary for
  successful real world use. This is particularly critical in public health,
  where failure to retrieve relevant, accurate, and current information could
  significantly impact UK residents. However, currently little is known about LLM
  knowledge of UK Government public health information. To address this issue,
  this paper introduces a new benchmark, PubHealthBench, with over 8000 questions
  for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form
  responses to public health queries, created via an automated pipeline. We also
  release a new dataset of the extracted UK Government public health guidance
  documents used as source text for PubHealthBench. Assessing 24 LLMs on
  PubHealthBench we find the latest private LLMs (GPT-4.5, GPT-4.1 and o1) have a
  high degree of knowledge, achieving >90% in the MCQA setup, and outperform
  humans with cursory search engine use. However, in the free form setup we see
  lower performance with no model scoring >75%. Therefore, whilst there are
  promising signs that state of the art (SOTA) LLMs are an increasingly accurate
  source of public health information, additional safeguards or tools may still
  be needed when providing free form responses on public health topics.
paper_ai_summary: A new benchmark, PubHealthBench, evaluates LLMs' knowledge of UK Government public health information, finding high accuracy in MCQA but lower performance in free form responses.
paper_ai_keywords: Large Language Models (LLMs) Multiple Choice Question Answering (MCQA) public health PubHealthBench GPT-4.5 GPT-4.1 o1 state of the art (SOTA) free form responses