title: Compression Represents Intelligence Linearly
summary: There is a belief that learning to compress well will lead to intelligence.
  Recently, language modeling has been shown to be equivalent to compression,
  which offers a compelling rationale for the success of large language models
  (LLMs): the development of more advanced language models is essentially
  enhancing compression which facilitates intelligence. Despite such appealing
  discussions, little empirical evidence is present for the interplay between
  compression and intelligence. In this work, we examine their relationship in
  the context of LLMs, treating LLMs as data compressors. Given the abstract
  concept of "intelligence", we adopt the average downstream benchmark scores as
  a surrogate, specifically targeting intelligence related to knowledge and
  commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our
  study brings together 30 public LLMs that originate from diverse organizations.
  Remarkably, we find that LLMs' intelligence -- reflected by average benchmark
  scores -- almost linearly correlates with their ability to compress external
  text corpora. These results provide concrete evidence supporting the belief
  that superior compression indicates greater intelligence. Furthermore, our
  findings suggest that compression efficiency, as an unsupervised metric derived
  from raw text corpora, serves as a reliable evaluation measure that is linearly
  associated with the model capabilities. We open-source our compression datasets
  as well as our data collection pipelines to facilitate future researchers to
  assess compression properly.
paper_authors._id: 661df263e5094183cd1008d8
paper_authors.name: Yuzhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T06:27:09.152Z
paper_authors._id: 6462def82a83863b97c0611e
paper_authors.avatarUrl: /avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
paper_authors.fullname: Yuzhen Huang
paper_authors.type: user
paper_authors.user: yuzhen17
paper_authors._id: 661df263e5094183cd1008d9
paper_authors.name: Jinghan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-18T19:35:11.700Z
paper_authors._id: 642c5631ab0cc792e4379099
paper_authors.avatarUrl: /avatars/c660650950d4804f48329970226e63d1.svg
paper_authors.fullname: Jinghan ZHANG
paper_authors.type: user
paper_authors.user: jinghan23
paper_authors._id: 661df263e5094183cd1008da
paper_authors.name: Zifei Shan
paper_authors._id: 661df263e5094183cd1008db
paper_authors.name: Junxian He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:42:16.631Z
paper_authors._id: 615f34ec3f6d24d67c1b5c78
paper_authors.avatarUrl: /avatars/6dcff6477993d9e57c5cb92b6f95eb66.svg
paper_authors.fullname: Junxian He
paper_authors.type: user
paper_authors.user: jxhe
paper_title: Compression Represents Intelligence Linearly
paper_summary: There is a belief that learning to compress well will lead to intelligence.
  Recently, language modeling has been shown to be equivalent to compression,
  which offers a compelling rationale for the success of large language models
  (LLMs): the development of more advanced language models is essentially
  enhancing compression which facilitates intelligence. Despite such appealing
  discussions, little empirical evidence is present for the interplay between
  compression and intelligence. In this work, we examine their relationship in
  the context of LLMs, treating LLMs as data compressors. Given the abstract
  concept of "intelligence", we adopt the average downstream benchmark scores as
  a surrogate, specifically targeting intelligence related to knowledge and
  commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our
  study brings together 30 public LLMs that originate from diverse organizations.
  Remarkably, we find that LLMs' intelligence -- reflected by average benchmark
  scores -- almost linearly correlates with their ability to compress external
  text corpora. These results provide concrete evidence supporting the belief
  that superior compression indicates greater intelligence. Furthermore, our
  findings suggest that compression efficiency, as an unsupervised metric derived
  from raw text corpora, serves as a reliable evaluation measure that is linearly
  associated with the model capabilities. We open-source our compression datasets
  as well as our data collection pipelines to facilitate future researchers to
  assess compression properly.
paper_ai_summary: Research shows that the ability of large language models to compress text is linearly associated with their performance on various benchmarks related to intelligence, suggesting compression efficiency as a reliable evaluation metric.
paper_ai_keywords: large language models compression benchmarks knowledge commonsense coding mathematical reasoning compression datasets