Law of the Weakest Link: Cross Capabilities of Large Language Models
The development and evaluation of Large Language Models (LLMs) have largely
focused on individual capabilities. However, this overlooks the intersection of
multiple abilities across different types of expertise that are often required
for real-world tasks, which we term cross capabilities. To systematically
explore this concept, we first define seven core individual capabilities and
then pair them to form seven common cross capabilities, each supported by a
manually constructed taxonomy. Building on these definitions, we introduce
CrossEval, a benchmark comprising 1,400 human-annotated prompts, with 100
prompts for each individual and cross capability. To ensure reliable
evaluation, we involve expert annotators to assess 4,200 model responses,
gathering 8,400 human ratings with detailed explanations to serve as reference
examples. Our findings reveal that, in both static evaluations and attempts to
enhance specific abilities, current LLMs consistently exhibit the "Law of the
Weakest Link," where cross-capability performance is significantly constrained
by the weakest component. Specifically, across 58 cross-capability scores from
17 models, 38 scores are lower than all individual capabilities, while 20 fall
between strong and weak, but closer to the weaker ability. These results
highlight the under-performance of LLMs in cross-capability tasks, making the
identification and improvement of the weakest capabilities a critical priority
for future research to optimize performance in complex, multi-dimensional
scenarios.
66fb784de66d0cc6ac37da70
Ming Zhong
claimed_verified
2024-10-02T07:41:26.655Z
61d53df2062444ea769d3b79
/avatars/fa771202368b6b2626a8fdf1c4369239.svg
Ming Zhong
user
MingZhong
66fb784de66d0cc6ac37da71
Aston Zhang
admin_assigned
2024-10-02T07:55:08.910Z
62e1bad44db2175cd277351a
/avatars/19451917f5edb201d378daca1c94cfd7.svg
Aston Zhang
user
astonzhang
66fb784de66d0cc6ac37da72
Xuewei Wang
admin_assigned
2024-10-02T07:55:14.816Z
66a331d9e42ce7eca24f0826
/avatars/a8fd5d17331ac32e9d994b3e3a2291e9.svg
Sherie
user
xwwang
66fb784de66d0cc6ac37da73
Rui Hou
admin_assigned
2024-10-02T07:55:20.417Z
65dbf0e0c2496f3ed8d39c68
/avatars/0d57f8cc656e57ba6c0edf2407d8d56e.svg
Rui Hou
user
ruihou16
66fb784de66d0cc6ac37da74
Wenhan Xiong
admin_assigned
2024-10-02T07:55:25.998Z
66a159e608767ced9f37f49c
/avatars/07d3ca897d0e1d6d467e220cd943ba3c.svg
Wenhan Xiong
user
xwhan
66fb784de66d0cc6ac37da75
Chenguang Zhu
66fb784de66d0cc6ac37da76
Zhengxing Chen
66fb784de66d0cc6ac37da77
Liang Tan
66fb784de66d0cc6ac37da78
Chloe Bi
admin_assigned
2024-10-02T07:56:08.480Z
64daa3a03ae9693d99e44858
/avatars/21c569fa30319387a70b7369bed73888.svg
Chloe Bi
user
chloebi98
66fb784de66d0cc6ac37da79
Mike Lewis
admin_assigned
2024-10-02T07:56:22.708Z
5e67eda71009063689407484
/avatars/6d7688faba888d463584eaa02ad011e3.svg
Mike Lewis
user
mikelewis0
66fb784de66d0cc6ac37da7a
Sravya Popuri
admin_assigned
2024-10-02T07:56:28.296Z
62e190f3926f4892a4c8099e
/avatars/2168f769c526362f9ed35473f4b30047.svg
Sravya Popuri
user
sravyapopuri388
66fb784de66d0cc6ac37da7b
Sharan Narang
66fb784de66d0cc6ac37da7c
Melanie Kambadur
66fb784de66d0cc6ac37da7d
Dhruv Mahajan
admin_assigned
2024-10-02T07:56:57.487Z
64358a51f81a16e7436330f7
/avatars/71c350836e5d025b4c7b9f99f72c2e21.svg
Dhruv Mahajan
user
Dhruv2009
66fb784de66d0cc6ac37da7e
Sergey Edunov
admin_assigned
2024-10-02T07:57:03.596Z
639a1b1857595ea3838b4ed7
/avatars/1cda3a496726dbf50ce0bc00ca1004a7.svg
Sergey Edunov
user
edunov
66fb784de66d0cc6ac37da7f
Jiawei Han
admin_assigned
2024-10-02T07:57:09.332Z
622533b1f3c0f2c4e741d9d2
https://cdn-avatars.huggingface.co/v1/production/uploads/1646605228784-noauth.png
Jiawei Han
user
alwaysraul
66fb784de66d0cc6ac37da80
Laurens van der Maaten
claimed_verified
2024-10-03T08:31:47.273Z
6311629dbc152fa3e8144e1e
/avatars/27f325beae3ee09f70fe4c341ca24790.svg
Laurens
user
lvdmaaten
Law of the Weakest Link: Cross Capabilities of Large Language Models
The development and evaluation of Large Language Models (LLMs) have largely
focused on individual capabilities. However, this overlooks the intersection of
multiple abilities across different types of expertise that are often required
for real-world tasks, which we term cross capabilities. To systematically
explore this concept, we first define seven core individual capabilities and
then pair them to form seven common cross capabilities, each supported by a
manually constructed taxonomy. Building on these definitions, we introduce
CrossEval, a benchmark comprising 1,400 human-annotated prompts, with 100
prompts for each individual and cross capability. To ensure reliable
evaluation, we involve expert annotators to assess 4,200 model responses,
gathering 8,400 human ratings with detailed explanations to serve as reference
examples. Our findings reveal that, in both static evaluations and attempts to
enhance specific abilities, current LLMs consistently exhibit the "Law of the
Weakest Link," where cross-capability performance is significantly constrained
by the weakest component. Specifically, across 58 cross-capability scores from
17 models, 38 scores are lower than all individual capabilities, while 20 fall
between strong and weak, but closer to the weaker ability. These results
highlight the under-performance of LLMs in cross-capability tasks, making the
identification and improvement of the weakest capabilities a critical priority
for future research to optimize performance in complex, multi-dimensional
scenarios.
LLMs show suboptimal performance in cross-capability tasks, constrained by their weakest individual capabilities, as demonstrated by the CrossEval benchmark.
Large Language Models CrossEval cross capabilities individual capabilities Law of the Weakest Link