title: GPT-Fathom: Benchmarking Large Language Models to Decipher the
    Evolutionary Path towards GPT-4 and Beyond
summary: With the rapid advancement of large language models (LLMs), there is a
  pressing need for a comprehensive evaluation suite to assess their capabilities
  and limitations. Existing LLM leaderboards often reference scores reported in
  other papers without consistent settings and prompts, which may inadvertently
  encourage cherry-picking favored settings and prompts for better results. In
  this work, we introduce GPT-Fathom, an open-source and reproducible LLM
  evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+
  leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across
  7 capability categories, all under aligned settings. Our retrospective study on
  OpenAI's earlier models offers valuable insights into the evolutionary path
  from GPT-3 to GPT-4. Currently, the community is eager to know how GPT-3
  progressively improves to GPT-4, including technical details like whether
  adding code data improves LLM's reasoning capability, which aspects of LLM
  capability can be improved by SFT and RLHF, how much is the alignment tax, etc.
  Our analysis sheds light on many of these questions, aiming to improve the
  transparency of advanced LLMs.
paper_authors._id: 65162feec1ec63624f568407
paper_authors.name: Shen Zheng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-29T02:02:20.295Z
paper_authors._id: 62bac112a10f784005d4881b
paper_authors.avatarUrl: /avatars/2e1cd972693a11b1b740f188f94e6122.svg
paper_authors.fullname: Shen Zheng
paper_authors.type: user
paper_authors.user: jasonzheng
paper_authors._id: 65162feec1ec63624f568408
paper_authors.name: Yuyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T12:11:33.704Z
paper_authors._id: 60dadd2c1f1eec955a46d059
paper_authors.avatarUrl: /avatars/6f551f1878b9b7ba59888048edd9c98d.svg
paper_authors.fullname: zhang
paper_authors.type: user
paper_authors.user: yuyu
paper_authors._id: 65162feec1ec63624f568409
paper_authors.name: Yijie Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T12:11:42.107Z
paper_authors._id: 63f005bb4a788ed1dd86faf4
paper_authors.avatarUrl: /avatars/fa79d0938f48c024f760a1c75ebadb40.svg
paper_authors.fullname: EJ Zhu
paper_authors.type: user
paper_authors.user: yijiezhu
paper_authors._id: 65162feec1ec63624f56840a
paper_authors.name: Chenguang Xi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-02T06:59:44.454Z
paper_authors._id: 6488e5e2013aacdc919252c0
paper_authors.avatarUrl: /avatars/f7c5756c29f28d0617f60cd48670296e.svg
paper_authors.fullname: Alex Xi
paper_authors.type: user
paper_authors.user: 7uckingMad
paper_authors._id: 65162feec1ec63624f56840b
paper_authors.name: Pengyang Gao
paper_authors._id: 65162feec1ec63624f56840c
paper_authors.name: Xun Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T12:12:09.935Z
paper_authors._id: 62533db4a06ec75172eeabe7
paper_authors.avatarUrl: /avatars/b1a4dad90afae5c00df97233a97777db.svg
paper_authors.fullname: xunzhou
paper_authors.type: user
paper_authors.user: xunzhou
paper_authors._id: 65162feec1ec63624f56840d
paper_authors.name: Kevin Chen-Chuan Chang
paper_title: GPT-Fathom: Benchmarking Large Language Models to Decipher the
    Evolutionary Path towards GPT-4 and Beyond
paper_summary: With the rapid advancement of large language models (LLMs), there is a
  pressing need for a comprehensive evaluation suite to assess their capabilities
  and limitations. Existing LLM leaderboards often reference scores reported in
  other papers without consistent settings and prompts, which may inadvertently
  encourage cherry-picking favored settings and prompts for better results. In
  this work, we introduce GPT-Fathom, an open-source and reproducible LLM
  evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+
  leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across
  7 capability categories, all under aligned settings. Our retrospective study on
  OpenAI's earlier models offers valuable insights into the evolutionary path
  from GPT-3 to GPT-4. Currently, the community is eager to know how GPT-3
  progressively improves to GPT-4, including technical details like whether
  adding code data improves LLM's reasoning capability, which aspects of LLM
  capability can be improved by SFT and RLHF, how much is the alignment tax, etc.
  Our analysis sheds light on many of these questions, aiming to improve the
  transparency of advanced LLMs.
paper_ai_summary: GPT-Fathom evaluates leading LLMs using a standardized suite to provide insights into the evolution from GPT-3 to GPT-4, addressing aspects like reasoning capabilities, SFT, RLHF, and alignment.
paper_ai_keywords: large language models LLM GPT-Fathom OpenAI Evals benchmarks capability categories systematic evaluation OpenAI's legacy models GPT-3 GPT-4 cherry-picking SFT RLHF alignment tax