title: Chain-of-Thought Hub: A Continuous Effort to Measure Large Language
    Models' Reasoning Performance
summary: As large language models (LLMs) are continuously being developed, their
  evaluation becomes increasingly important yet challenging. This work proposes
  Chain-of-Thought Hub, an open-source evaluation suite on the multi-step
  reasoning capabilities of large language models. We are interested in this
  setting for two reasons: (1) from the behavior of GPT and PaLM model family, we
  observe that complex reasoning is likely to be a key differentiator between
  weaker and stronger LLMs; (2) we envisage large language models to become the
  next-generation computational platform and foster an ecosystem of LLM-based new
  applications, this naturally requires the foundation models to perform complex
  tasks that often involve the composition of linguistic and logical operations.
  Our approach is to compile a suite of challenging reasoning benchmarks to track
  the progress of LLMs. Our current results show that: (1) model scale clearly
  correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and
  PaLM-2 are the only two models that are comparable with GPT-4, while
  open-sourced models still lag behind; (3) LLaMA-65B performs closely to
  code-davinci-002, indicating that with successful further development such as
  reinforcement learning from human feedback (RLHF), it has great potential to be
  close to GPT-3.5-Turbo. Our results also suggest that for the open-source
  efforts to catch up, the community may focus more on building better base
  models and exploring RLHF.
paper_authors._id: 64755ba1a51711a3b5910265
paper_authors.name: Yao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T11:11:30.920Z
paper_authors._id: 636c929d5ef5a1a07db4e785
paper_authors.avatarUrl: /avatars/1bb48dbba964f7fa84036586032684cf.svg
paper_authors.fullname: Yao Fu
paper_authors.type: user
paper_authors.user: yaofu
paper_authors._id: 64755ba1a51711a3b5910266
paper_authors.name: Litu Ou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T05:54:20.066Z
paper_authors._id: 622f2feea32d46b4be9ed8c4
paper_authors.avatarUrl: /avatars/ba25eb941a7c9a414b7fd4818adfa26b.svg
paper_authors.fullname: Litu Ou
paper_authors.type: user
paper_authors.user: learn3r
paper_authors._id: 64755ba1a51711a3b5910267
paper_authors.name: Mingyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-06T09:36:27.070Z
paper_authors._id: 64a5d3802a782c381a72cae6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a5d3802a782c381a72cae6/LqCm0aA7Q6-qLCUhxfiZ8.png
paper_authors.fullname: Alex
paper_authors.type: user
paper_authors.user: iPruning
paper_authors._id: 64755ba1a51711a3b5910268
paper_authors.name: Yuhao Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T16:32:21.887Z
paper_authors._id: 643ae8513b71e2b0cc1e4c8c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643ae8513b71e2b0cc1e4c8c/IFhHi8bIaBXnu82cPDBA3.jpeg
paper_authors.fullname: Yuhao Wan
paper_authors.type: user
paper_authors.user: dianewan
paper_authors._id: 64755ba1a51711a3b5910269
paper_authors.name: Hao Peng
paper_authors._id: 64755ba1a51711a3b591026a
paper_authors.name: Tushar Khot
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T07:24:30.272Z
paper_authors._id: 62cdf2dbaac2c91c95581830
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
paper_authors.fullname: Tushar Khot
paper_authors.type: user
paper_authors.user: tusharkhot
paper_title: Chain-of-Thought Hub: A Continuous Effort to Measure Large Language
    Models' Reasoning Performance
paper_summary: As large language models (LLMs) are continuously being developed, their
  evaluation becomes increasingly important yet challenging. This work proposes
  Chain-of-Thought Hub, an open-source evaluation suite on the multi-step
  reasoning capabilities of large language models. We are interested in this
  setting for two reasons: (1) from the behavior of GPT and PaLM model family, we
  observe that complex reasoning is likely to be a key differentiator between
  weaker and stronger LLMs; (2) we envisage large language models to become the
  next-generation computational platform and foster an ecosystem of LLM-based new
  applications, this naturally requires the foundation models to perform complex
  tasks that often involve the composition of linguistic and logical operations.
  Our approach is to compile a suite of challenging reasoning benchmarks to track
  the progress of LLMs. Our current results show that: (1) model scale clearly
  correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and
  PaLM-2 are the only two models that are comparable with GPT-4, while
  open-sourced models still lag behind; (3) LLaMA-65B performs closely to
  code-davinci-002, indicating that with successful further development such as
  reinforcement learning from human feedback (RLHF), it has great potential to be
  close to GPT-3.5-Turbo. Our results also suggest that for the open-source
  efforts to catch up, the community may focus more on building better base
  models and exploring RLHF.
paper_ai_summary: Chain-of-Thought Hub evaluates multi-step reasoning capabilities of large language models using a suite of benchmarks, showing model scale correlates with reasoning ability and highlighting the need for improvements in base models and reinforcement learning from human feedback for open-source models.
paper_ai_keywords: large language models multi-step reasoning Chain-of-Thought Hub reasoning benchmarks reinforcement learning from human feedback