LightThinker: Thinking Step-by-Step Compression
Large language models (LLMs) have shown remarkable performance in complex
reasoning tasks, but their efficiency is hindered by the substantial memory and
computational costs associated with generating lengthy tokens. In this paper,
we propose LightThinker, a novel method that enables LLMs to dynamically
compress intermediate thoughts during reasoning. Inspired by human cognitive
processes, LightThinker compresses verbose thought steps into compact
representations and discards the original reasoning chains, thereby
significantly reducing the number of tokens stored in the context window. This
is achieved by training the model on when and how to perform compression
through data construction, mapping hidden states to condensed gist tokens, and
creating specialized attention masks. Additionally, we introduce the Dependency
(Dep) metric to quantify the degree of compression by measuring the reliance on
historical tokens during generation. Extensive experiments on four datasets and
two models show that LightThinker reduces peak memory usage and inference time,
while maintaining competitive accuracy. Our work provides a new direction for
improving the efficiency of LLMs in complex reasoning tasks without sacrificing
performance. Code will be released at https://github.com/zjunlp/LightThinker.
67bbfe2d670ece8d9184f339
Jintian Zhang
admin_assigned
2025-02-24T13:22:07.730Z
63f89fe7565506a9cadcd2cf
/avatars/7eb449a1109dcff051cb3ba680f0c082.svg
Jintian Zhang
user
MikeDean
67bbfe2d670ece8d9184f33a
Yuqi Zhu
claimed_verified
2025-06-25T13:33:00.564Z
679b6d152583e6629337c592
/avatars/7ad77d7384fd41ad987b6eb5907a8f08.svg
Yuqi Zhu
user
Yukirsh
67bbfe2d670ece8d9184f33b
Mengshu Sun
admin_assigned
2025-02-24T13:22:36.798Z
64f6d1ed46284aa28d9abf6c
/avatars/d6beaecfd00345e4a664862fff217427.svg
Sun mengshu
user
sunmengshu
67bbfe2d670ece8d9184f33c
Yujie Luo
admin_assigned
2025-02-24T13:22:42.818Z
67603f17d4f2ded0c1498358
/avatars/97205504757d7eb33512ab96b2ecde28.svg
yujieluo
user
yujieluo1031
67bbfe2d670ece8d9184f33d
Shuofei Qiao
claimed_verified
2025-02-24T09:07:02.722Z
6447800f30fa4ecb85ddad80
https://cdn-avatars.huggingface.co/v1/production/uploads/6447800f30fa4ecb85ddad80/NsmXIaMsWctmTNA7tFVkX.jpeg
Shuofei Qiao
user
GoooDte
67bbfe2d670ece8d9184f33e
Lun Du
67bbfe2d670ece8d9184f33f
Da Zheng
admin_assigned
2025-02-24T13:22:53.988Z
66270e0026d5a3eee310ad53
/avatars/db34068c114c348de296e00b1b5a5b9b.svg
Da Zheng
user
zhengda1936
67bbfe2d670ece8d9184f340
Huajun Chen
admin_assigned
2025-02-24T13:24:31.548Z
64931296137833d7ec7689cd
https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
Huajun Chen
user
huajunsir
67bbfe2d670ece8d9184f341
Ningyu Zhang
claimed_verified
2025-02-24T09:07:04.794Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
LightThinker: Thinking Step-by-Step Compression
Large language models (LLMs) have shown remarkable performance in complex
reasoning tasks, but their efficiency is hindered by the substantial memory and
computational costs associated with generating lengthy tokens. In this paper,
we propose LightThinker, a novel method that enables LLMs to dynamically
compress intermediate thoughts during reasoning. Inspired by human cognitive
processes, LightThinker compresses verbose thought steps into compact
representations and discards the original reasoning chains, thereby
significantly reducing the number of tokens stored in the context window. This
is achieved by training the model on when and how to perform compression
through data construction, mapping hidden states to condensed gist tokens, and
creating specialized attention masks. Additionally, we introduce the Dependency
(Dep) metric to quantify the degree of compression by measuring the reliance on
historical tokens during generation. Extensive experiments on four datasets and
two models show that LightThinker reduces peak memory usage and inference time,
while maintaining competitive accuracy. Our work provides a new direction for
improving the efficiency of LLMs in complex reasoning tasks without sacrificing
performance. Code will be released at https://github.com/zjunlp/LightThinker.
LightThinker improves efficiency in LLMs by dynamically compressing intermediate thoughts, reducing memory usage and inference time while maintaining performance.
Large language models LightThinker intermediate thoughts memory costs computational costs token compression context window data construction hidden states gist tokens attention masks Dependency metric complex reasoning tasks