Chain-of-Thought Hub: A Continuous Effort to Measure Large Language
  Models' Reasoning Performance
As large language models (LLMs) are continuously being developed, their
evaluation becomes increasingly important yet challenging. This work proposes
Chain-of-Thought Hub, an open-source evaluation suite on the multi-step
reasoning capabilities of large language models. We are interested in this
setting for two reasons: (1) from the behavior of GPT and PaLM model family, we
observe that complex reasoning is likely to be a key differentiator between
weaker and stronger LLMs; (2) we envisage large language models to become the
next-generation computational platform and foster an ecosystem of LLM-based new
applications, this naturally requires the foundation models to perform complex
tasks that often involve the composition of linguistic and logical operations.
Our approach is to compile a suite of challenging reasoning benchmarks to track
the progress of LLMs. Our current results show that: (1) model scale clearly
correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and
PaLM-2 are the only two models that are comparable with GPT-4, while
open-sourced models still lag behind; (3) LLaMA-65B performs closely to
code-davinci-002, indicating that with successful further development such as
reinforcement learning from human feedback (RLHF), it has great potential to be
close to GPT-3.5-Turbo. Our results also suggest that for the open-source
efforts to catch up, the community may focus more on building better base
models and exploring RLHF.
64755ba1a51711a3b5910265
Yao Fu
claimed_verified
2023-05-30T11:11:30.920Z
636c929d5ef5a1a07db4e785
/avatars/1bb48dbba964f7fa84036586032684cf.svg
Yao Fu
user
yaofu
64755ba1a51711a3b5910266
Litu Ou
claimed_verified
2023-06-01T05:54:20.066Z
622f2feea32d46b4be9ed8c4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NDeZQZQK5U-9m10yQwDVf.png
Litu Ou
user
learn3r
64755ba1a51711a3b5910267
Mingyu Chen
claimed_verified
2023-07-06T09:36:27.070Z
64a5d3802a782c381a72cae6
https://cdn-avatars.huggingface.co/v1/production/uploads/64a5d3802a782c381a72cae6/LqCm0aA7Q6-qLCUhxfiZ8.png
Alex
user
iPruning
64755ba1a51711a3b5910268
Yuhao Wan
claimed_verified
2023-06-01T16:32:21.887Z
643ae8513b71e2b0cc1e4c8c
https://cdn-avatars.huggingface.co/v1/production/uploads/643ae8513b71e2b0cc1e4c8c/IFhHi8bIaBXnu82cPDBA3.jpeg
Yuhao Wan
user
dianewan
64755ba1a51711a3b5910269
Hao Peng
64755ba1a51711a3b591026a
Tushar Khot
claimed_verified
2023-05-31T07:24:30.272Z
62cdf2dbaac2c91c95581830
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
Tushar Khot
user
tusharkhot
Chain-of-Thought Hub: A Continuous Effort to Measure Large Language
  Models' Reasoning Performance
As large language models (LLMs) are continuously being developed, their
evaluation becomes increasingly important yet challenging. This work proposes
Chain-of-Thought Hub, an open-source evaluation suite on the multi-step
reasoning capabilities of large language models. We are interested in this
setting for two reasons: (1) from the behavior of GPT and PaLM model family, we
observe that complex reasoning is likely to be a key differentiator between
weaker and stronger LLMs; (2) we envisage large language models to become the
next-generation computational platform and foster an ecosystem of LLM-based new
applications, this naturally requires the foundation models to perform complex
tasks that often involve the composition of linguistic and logical operations.
Our approach is to compile a suite of challenging reasoning benchmarks to track
the progress of LLMs. Our current results show that: (1) model scale clearly
correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and
PaLM-2 are the only two models that are comparable with GPT-4, while
open-sourced models still lag behind; (3) LLaMA-65B performs closely to
code-davinci-002, indicating that with successful further development such as
reinforcement learning from human feedback (RLHF), it has great potential to be
close to GPT-3.5-Turbo. Our results also suggest that for the open-source
efforts to catch up, the community may focus more on building better base
models and exploring RLHF.
Chain-of-Thought Hub evaluates multi-step reasoning capabilities of large language models using a suite of benchmarks, showing model scale correlates with reasoning ability and highlighting the need for improvements in base models and reinforcement learning from human feedback for open-source models.
large language models multi-step reasoning Chain-of-Thought Hub reasoning benchmarks reinforcement learning from human feedback