title: LightThinker: Thinking Step-by-Step Compression
summary: Large language models (LLMs) have shown remarkable performance in complex
  reasoning tasks, but their efficiency is hindered by the substantial memory and
  computational costs associated with generating lengthy tokens. In this paper,
  we propose LightThinker, a novel method that enables LLMs to dynamically
  compress intermediate thoughts during reasoning. Inspired by human cognitive
  processes, LightThinker compresses verbose thought steps into compact
  representations and discards the original reasoning chains, thereby
  significantly reducing the number of tokens stored in the context window. This
  is achieved by training the model on when and how to perform compression
  through data construction, mapping hidden states to condensed gist tokens, and
  creating specialized attention masks. Additionally, we introduce the Dependency
  (Dep) metric to quantify the degree of compression by measuring the reliance on
  historical tokens during generation. Extensive experiments on four datasets and
  two models show that LightThinker reduces peak memory usage and inference time,
  while maintaining competitive accuracy. Our work provides a new direction for
  improving the efficiency of LLMs in complex reasoning tasks without sacrificing
  performance. Code will be released at https://github.com/zjunlp/LightThinker.
paper_authors._id: 67bbfe2d670ece8d9184f339
paper_authors.name: Jintian Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:22:07.730Z
paper_authors._id: 63f89fe7565506a9cadcd2cf
paper_authors.avatarUrl: /avatars/7eb449a1109dcff051cb3ba680f0c082.svg
paper_authors.fullname: Jintian Zhang
paper_authors.type: user
paper_authors.user: MikeDean
paper_authors._id: 67bbfe2d670ece8d9184f33a
paper_authors.name: Yuqi Zhu
paper_authors._id: 67bbfe2d670ece8d9184f33b
paper_authors.name: Mengshu Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:22:36.798Z
paper_authors._id: 64f6d1ed46284aa28d9abf6c
paper_authors.avatarUrl: /avatars/d6beaecfd00345e4a664862fff217427.svg
paper_authors.fullname: Sun mengshu
paper_authors.type: user
paper_authors.user: sunmengshu
paper_authors._id: 67bbfe2d670ece8d9184f33c
paper_authors.name: Yujie Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:22:42.818Z
paper_authors._id: 67603f17d4f2ded0c1498358
paper_authors.avatarUrl: /avatars/97205504757d7eb33512ab96b2ecde28.svg
paper_authors.fullname: yujieluo
paper_authors.type: user
paper_authors.user: yujieluo1031
paper_authors._id: 67bbfe2d670ece8d9184f33d
paper_authors.name: Shuofei Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:07:02.722Z
paper_authors._id: 6447800f30fa4ecb85ddad80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6447800f30fa4ecb85ddad80/NsmXIaMsWctmTNA7tFVkX.jpeg
paper_authors.fullname: Shuofei Qiao
paper_authors.type: user
paper_authors.user: GoooDte
paper_authors._id: 67bbfe2d670ece8d9184f33e
paper_authors.name: Lun Du
paper_authors._id: 67bbfe2d670ece8d9184f33f
paper_authors.name: Da Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:22:53.988Z
paper_authors._id: 66270e0026d5a3eee310ad53
paper_authors.avatarUrl: /avatars/db34068c114c348de296e00b1b5a5b9b.svg
paper_authors.fullname: Da Zheng
paper_authors.type: user
paper_authors.user: zhengda1936
paper_authors._id: 67bbfe2d670ece8d9184f340
paper_authors.name: Huajun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:24:31.548Z
paper_authors._id: 64931296137833d7ec7689cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
paper_authors.fullname: Huajun Chen
paper_authors.type: user
paper_authors.user: huajunsir
paper_authors._id: 67bbfe2d670ece8d9184f341
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:07:04.794Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_title: LightThinker: Thinking Step-by-Step Compression
paper_summary: Large language models (LLMs) have shown remarkable performance in complex
  reasoning tasks, but their efficiency is hindered by the substantial memory and
  computational costs associated with generating lengthy tokens. In this paper,
  we propose LightThinker, a novel method that enables LLMs to dynamically
  compress intermediate thoughts during reasoning. Inspired by human cognitive
  processes, LightThinker compresses verbose thought steps into compact
  representations and discards the original reasoning chains, thereby
  significantly reducing the number of tokens stored in the context window. This
  is achieved by training the model on when and how to perform compression
  through data construction, mapping hidden states to condensed gist tokens, and
  creating specialized attention masks. Additionally, we introduce the Dependency
  (Dep) metric to quantify the degree of compression by measuring the reliance on
  historical tokens during generation. Extensive experiments on four datasets and
  two models show that LightThinker reduces peak memory usage and inference time,
  while maintaining competitive accuracy. Our work provides a new direction for
  improving the efficiency of LLMs in complex reasoning tasks without sacrificing
  performance. Code will be released at https://github.com/zjunlp/LightThinker.
paper_ai_summary: LightThinker improves efficiency in LLMs by dynamically compressing intermediate thoughts, reducing memory usage and inference time while maintaining performance.
paper_ai_keywords: Large language models LightThinker intermediate thoughts memory costs computational costs token compression context window data construction hidden states gist tokens attention masks Dependency metric complex reasoning tasks