title: Law of the Weakest Link: Cross Capabilities of Large Language Models
summary: The development and evaluation of Large Language Models (LLMs) have largely
  focused on individual capabilities. However, this overlooks the intersection of
  multiple abilities across different types of expertise that are often required
  for real-world tasks, which we term cross capabilities. To systematically
  explore this concept, we first define seven core individual capabilities and
  then pair them to form seven common cross capabilities, each supported by a
  manually constructed taxonomy. Building on these definitions, we introduce
  CrossEval, a benchmark comprising 1,400 human-annotated prompts, with 100
  prompts for each individual and cross capability. To ensure reliable
  evaluation, we involve expert annotators to assess 4,200 model responses,
  gathering 8,400 human ratings with detailed explanations to serve as reference
  examples. Our findings reveal that, in both static evaluations and attempts to
  enhance specific abilities, current LLMs consistently exhibit the "Law of the
  Weakest Link," where cross-capability performance is significantly constrained
  by the weakest component. Specifically, across 58 cross-capability scores from
  17 models, 38 scores are lower than all individual capabilities, while 20 fall
  between strong and weak, but closer to the weaker ability. These results
  highlight the under-performance of LLMs in cross-capability tasks, making the
  identification and improvement of the weakest capabilities a critical priority
  for future research to optimize performance in complex, multi-dimensional
  scenarios.
paper_authors._id: 66fb784de66d0cc6ac37da70
paper_authors.name: Ming Zhong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T07:41:26.655Z
paper_authors._id: 61d53df2062444ea769d3b79
paper_authors.avatarUrl: /avatars/fa771202368b6b2626a8fdf1c4369239.svg
paper_authors.fullname: Ming Zhong
paper_authors.type: user
paper_authors.user: MingZhong
paper_authors._id: 66fb784de66d0cc6ac37da71
paper_authors.name: Aston Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:55:08.910Z
paper_authors._id: 62e1bad44db2175cd277351a
paper_authors.avatarUrl: /avatars/19451917f5edb201d378daca1c94cfd7.svg
paper_authors.fullname: Aston Zhang
paper_authors.type: user
paper_authors.user: astonzhang
paper_authors._id: 66fb784de66d0cc6ac37da72
paper_authors.name: Xuewei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:55:14.816Z
paper_authors._id: 66a331d9e42ce7eca24f0826
paper_authors.avatarUrl: /avatars/a8fd5d17331ac32e9d994b3e3a2291e9.svg
paper_authors.fullname: Sherie
paper_authors.type: user
paper_authors.user: xwwang
paper_authors._id: 66fb784de66d0cc6ac37da73
paper_authors.name: Rui Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:55:20.417Z
paper_authors._id: 65dbf0e0c2496f3ed8d39c68
paper_authors.avatarUrl: /avatars/0d57f8cc656e57ba6c0edf2407d8d56e.svg
paper_authors.fullname: Rui Hou
paper_authors.type: user
paper_authors.user: ruihou16
paper_authors._id: 66fb784de66d0cc6ac37da74
paper_authors.name: Wenhan Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:55:25.998Z
paper_authors._id: 66a159e608767ced9f37f49c
paper_authors.avatarUrl: /avatars/07d3ca897d0e1d6d467e220cd943ba3c.svg
paper_authors.fullname: Wenhan Xiong
paper_authors.type: user
paper_authors.user: xwhan
paper_authors._id: 66fb784de66d0cc6ac37da75
paper_authors.name: Chenguang Zhu
paper_authors._id: 66fb784de66d0cc6ac37da76
paper_authors.name: Zhengxing Chen
paper_authors._id: 66fb784de66d0cc6ac37da77
paper_authors.name: Liang Tan
paper_authors._id: 66fb784de66d0cc6ac37da78
paper_authors.name: Chloe Bi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:56:08.480Z
paper_authors._id: 64daa3a03ae9693d99e44858
paper_authors.avatarUrl: /avatars/21c569fa30319387a70b7369bed73888.svg
paper_authors.fullname: Chloe Bi
paper_authors.type: user
paper_authors.user: chloebi98
paper_authors._id: 66fb784de66d0cc6ac37da79
paper_authors.name: Mike Lewis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:56:22.708Z
paper_authors._id: 5e67eda71009063689407484
paper_authors.avatarUrl: /avatars/6d7688faba888d463584eaa02ad011e3.svg
paper_authors.fullname: Mike Lewis
paper_authors.type: user
paper_authors.user: mikelewis0
paper_authors._id: 66fb784de66d0cc6ac37da7a
paper_authors.name: Sravya Popuri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:56:28.296Z
paper_authors._id: 62e190f3926f4892a4c8099e
paper_authors.avatarUrl: /avatars/2168f769c526362f9ed35473f4b30047.svg
paper_authors.fullname: Sravya Popuri
paper_authors.type: user
paper_authors.user: sravyapopuri388
paper_authors._id: 66fb784de66d0cc6ac37da7b
paper_authors.name: Sharan Narang
paper_authors._id: 66fb784de66d0cc6ac37da7c
paper_authors.name: Melanie Kambadur
paper_authors._id: 66fb784de66d0cc6ac37da7d
paper_authors.name: Dhruv Mahajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:56:57.487Z
paper_authors._id: 64358a51f81a16e7436330f7
paper_authors.avatarUrl: /avatars/71c350836e5d025b4c7b9f99f72c2e21.svg
paper_authors.fullname: Dhruv Mahajan
paper_authors.type: user
paper_authors.user: Dhruv2009
paper_authors._id: 66fb784de66d0cc6ac37da7e
paper_authors.name: Sergey Edunov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:57:03.596Z
paper_authors._id: 639a1b1857595ea3838b4ed7
paper_authors.avatarUrl: /avatars/1cda3a496726dbf50ce0bc00ca1004a7.svg
paper_authors.fullname: Sergey Edunov
paper_authors.type: user
paper_authors.user: edunov
paper_authors._id: 66fb784de66d0cc6ac37da7f
paper_authors.name: Jiawei Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T07:57:09.332Z
paper_authors._id: 622533b1f3c0f2c4e741d9d2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1646605228784-noauth.png
paper_authors.fullname: Jiawei Han
paper_authors.type: user
paper_authors.user: alwaysraul
paper_authors._id: 66fb784de66d0cc6ac37da80
paper_authors.name: Laurens van der Maaten
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T08:31:47.273Z
paper_authors._id: 6311629dbc152fa3e8144e1e
paper_authors.avatarUrl: /avatars/27f325beae3ee09f70fe4c341ca24790.svg
paper_authors.fullname: Laurens
paper_authors.type: user
paper_authors.user: lvdmaaten
paper_title: Law of the Weakest Link: Cross Capabilities of Large Language Models
paper_summary: The development and evaluation of Large Language Models (LLMs) have largely
  focused on individual capabilities. However, this overlooks the intersection of
  multiple abilities across different types of expertise that are often required
  for real-world tasks, which we term cross capabilities. To systematically
  explore this concept, we first define seven core individual capabilities and
  then pair them to form seven common cross capabilities, each supported by a
  manually constructed taxonomy. Building on these definitions, we introduce
  CrossEval, a benchmark comprising 1,400 human-annotated prompts, with 100
  prompts for each individual and cross capability. To ensure reliable
  evaluation, we involve expert annotators to assess 4,200 model responses,
  gathering 8,400 human ratings with detailed explanations to serve as reference
  examples. Our findings reveal that, in both static evaluations and attempts to
  enhance specific abilities, current LLMs consistently exhibit the "Law of the
  Weakest Link," where cross-capability performance is significantly constrained
  by the weakest component. Specifically, across 58 cross-capability scores from
  17 models, 38 scores are lower than all individual capabilities, while 20 fall
  between strong and weak, but closer to the weaker ability. These results
  highlight the under-performance of LLMs in cross-capability tasks, making the
  identification and improvement of the weakest capabilities a critical priority
  for future research to optimize performance in complex, multi-dimensional
  scenarios.