GPT-Fathom: Benchmarking Large Language Models to Decipher the
  Evolutionary Path towards GPT-4 and Beyond
With the rapid advancement of large language models (LLMs), there is a
pressing need for a comprehensive evaluation suite to assess their capabilities
and limitations. Existing LLM leaderboards often reference scores reported in
other papers without consistent settings and prompts, which may inadvertently
encourage cherry-picking favored settings and prompts for better results. In
this work, we introduce GPT-Fathom, an open-source and reproducible LLM
evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+
leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across
7 capability categories, all under aligned settings. Our retrospective study on
OpenAI's earlier models offers valuable insights into the evolutionary path
from GPT-3 to GPT-4. Currently, the community is eager to know how GPT-3
progressively improves to GPT-4, including technical details like whether
adding code data improves LLM's reasoning capability, which aspects of LLM
capability can be improved by SFT and RLHF, how much is the alignment tax, etc.
Our analysis sheds light on many of these questions, aiming to improve the
transparency of advanced LLMs.
65162feec1ec63624f568407
Shen Zheng
extracted_confirmed
2023-09-29T02:02:20.295Z
62bac112a10f784005d4881b
/avatars/2e1cd972693a11b1b740f188f94e6122.svg
Shen Zheng
user
jasonzheng
65162feec1ec63624f568408
Yuyu Zhang
admin_assigned
2023-09-29T12:11:33.704Z
60dadd2c1f1eec955a46d059
/avatars/6f551f1878b9b7ba59888048edd9c98d.svg
zhang
user
yuyu
65162feec1ec63624f568409
Yijie Zhu
admin_assigned
2023-09-29T12:11:42.107Z
63f005bb4a788ed1dd86faf4
/avatars/fa79d0938f48c024f760a1c75ebadb40.svg
EJ Zhu
user
yijiezhu
65162feec1ec63624f56840a
Chenguang Xi
claimed_verified
2023-10-02T06:59:44.454Z
6488e5e2013aacdc919252c0
/avatars/f7c5756c29f28d0617f60cd48670296e.svg
Alex Xi
user
7uckingMad
65162feec1ec63624f56840b
Pengyang Gao
65162feec1ec63624f56840c
Xun Zhou
admin_assigned
2023-09-29T12:12:09.935Z
62533db4a06ec75172eeabe7
/avatars/b1a4dad90afae5c00df97233a97777db.svg
xunzhou
user
xunzhou
65162feec1ec63624f56840d
Kevin Chen-Chuan Chang
GPT-Fathom: Benchmarking Large Language Models to Decipher the
  Evolutionary Path towards GPT-4 and Beyond
With the rapid advancement of large language models (LLMs), there is a
pressing need for a comprehensive evaluation suite to assess their capabilities
and limitations. Existing LLM leaderboards often reference scores reported in
other papers without consistent settings and prompts, which may inadvertently
encourage cherry-picking favored settings and prompts for better results. In
this work, we introduce GPT-Fathom, an open-source and reproducible LLM
evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+
leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across
7 capability categories, all under aligned settings. Our retrospective study on
OpenAI's earlier models offers valuable insights into the evolutionary path
from GPT-3 to GPT-4. Currently, the community is eager to know how GPT-3
progressively improves to GPT-4, including technical details like whether
adding code data improves LLM's reasoning capability, which aspects of LLM
capability can be improved by SFT and RLHF, how much is the alignment tax, etc.
Our analysis sheds light on many of these questions, aiming to improve the
transparency of advanced LLMs.
GPT-Fathom evaluates leading LLMs using a standardized suite to provide insights into the evolution from GPT-3 to GPT-4, addressing aspects like reasoning capabilities, SFT, RLHF, and alignment.
large language models LLM GPT-Fathom OpenAI Evals benchmarks capability categories systematic evaluation OpenAI's legacy models GPT-3 GPT-4 cherry-picking SFT RLHF alignment tax