title: Critical Tokens Matter: Token-Level Contrastive Estimation Enhence LLM's
    Reasoning Capability
summary: Large Language Models (LLMs) have exhibited remarkable performance on
  reasoning tasks. They utilize autoregressive token generation to construct
  reasoning trajectories, enabling the development of a coherent chain of
  thought. In this work, we explore the impact of individual tokens on the final
  outcomes of reasoning tasks. We identify the existence of ``critical tokens''
  that lead to incorrect reasoning trajectories in LLMs. Specifically, we find
  that LLMs tend to produce positive outcomes when forced to decode other tokens
  instead of critical tokens. Motivated by this observation, we propose a novel
  approach - cDPO - designed to automatically recognize and conduct token-level
  rewards for the critical tokens during the alignment process. Specifically, we
  develop a contrastive estimation approach to automatically identify critical
  tokens. It is achieved by comparing the generation likelihood of positive and
  negative models. To achieve this, we separately fine-tune the positive and
  negative models on various reasoning trajectories, consequently, they are
  capable of identifying identify critical tokens within incorrect trajectories
  that contribute to erroneous outcomes. Moreover, to further align the model
  with the critical token information during the alignment process, we extend the
  conventional DPO algorithms to token-level DPO and utilize the differential
  likelihood from the aforementioned positive and negative model as important
  weight for token-level DPO learning.Experimental results on GSM8K and MATH500
  benchmarks with two-widely used models Llama-3 (8B and 70B) and deepseek-math
  (7B) demonstrate the effectiveness of the propsoed approach cDPO.
paper_authors._id: 674d9e3c0912639dc496ae7e
paper_authors.name: Zicheng Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T09:09:54.784Z
paper_authors._id: 64292eb375bcc24c5e52c011
paper_authors.avatarUrl: /avatars/c8cb03ca35ca12d8831be5f4e8547d54.svg
paper_authors.fullname: czl
paper_authors.type: user
paper_authors.user: Lin1557
paper_authors._id: 674d9e3c0912639dc496ae7f
paper_authors.name: Tian Liang
paper_authors._id: 674d9e3c0912639dc496ae80
paper_authors.name: Jiahao Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:55:17.273Z
paper_authors._id: 660399710f1fc2f16de18072
paper_authors.avatarUrl: /avatars/c22a749cc45db693c2d9ea877c7cace4.svg
paper_authors.fullname: Jiahao Xu
paper_authors.type: user
paper_authors.user: Jiahao004
paper_authors._id: 674d9e3c0912639dc496ae81
paper_authors.name: Xing Wang
paper_authors._id: 674d9e3c0912639dc496ae82
paper_authors.name: Ruilin Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-12T19:57:51.228Z
paper_authors._id: 6548956fe49bd8d58e8adf0e
paper_authors.avatarUrl: /avatars/24c54b14c253c87d4b7438193f16ce28.svg
paper_authors.fullname: Ruilin
paper_authors.type: user
paper_authors.user: Antimage01
paper_authors._id: 674d9e3c0912639dc496ae83
paper_authors.name: Chufan Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:54:37.406Z
paper_authors._id: 6373b559c3b104ef179cf503
paper_authors.avatarUrl: /avatars/36f36eea72a100ee56b37c6d185b47c0.svg
paper_authors.fullname: Shi
paper_authors.type: user
paper_authors.user: Chufan
paper_authors._id: 674d9e3c0912639dc496ae84
paper_authors.name: Siheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:54:27.900Z
paper_authors._id: 62f71b23d3bdacb7eec43b1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f71b23d3bdacb7eec43b1f/6RNVG9zug9tutjo5i8T4C.jpeg
paper_authors.fullname: SihengLi
paper_authors.type: user
paper_authors.user: SihengLi
paper_authors._id: 674d9e3c0912639dc496ae85
paper_authors.name: Yujiu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:54:21.597Z
paper_authors._id: 64ca1fe838837b12d5e529b7
paper_authors.avatarUrl: /avatars/44a3ad9e59318784ac531993b5f69f6b.svg
paper_authors.fullname: Yujiu Yang
paper_authors.type: user
paper_authors.user: Thu-redrobot
paper_authors._id: 674d9e3c0912639dc496ae86
paper_authors.name: Zhaopeng Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:54:15.917Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_title: Critical Tokens Matter: Token-Level Contrastive Estimation Enhence LLM's
    Reasoning Capability
paper_summary: Large Language Models (LLMs) have exhibited remarkable performance on
  reasoning tasks. They utilize autoregressive token generation to construct
  reasoning trajectories, enabling the development of a coherent chain of
  thought. In this work, we explore the impact of individual tokens on the final
  outcomes of reasoning tasks. We identify the existence of ``critical tokens''
  that lead to incorrect reasoning trajectories in LLMs. Specifically, we find
  that LLMs tend to produce positive outcomes when forced to decode other tokens
  instead of critical tokens. Motivated by this observation, we propose a novel
  approach - cDPO - designed to automatically recognize and conduct token-level
  rewards for the critical tokens during the alignment process. Specifically, we
  develop a contrastive estimation approach to automatically identify critical
  tokens. It is achieved by comparing the generation likelihood of positive and
  negative models. To achieve this, we separately fine-tune the positive and
  negative models on various reasoning trajectories, consequently, they are
  capable of identifying identify critical tokens within incorrect trajectories
  that contribute to erroneous outcomes. Moreover, to further align the model
  with the critical token information during the alignment process, we extend the
  conventional DPO algorithms to token-level DPO and utilize the differential
  likelihood from the aforementioned positive and negative model as important
  weight for token-level DPO learning.Experimental results on GSM8K and MATH500
  benchmarks with two-widely used models Llama-3 (8B and 70B) and deepseek-math
  (7B) demonstrate the effectiveness of the propsoed approach cDPO.
paper_ai_keywords: autoregressive token generation reasoning trajectories critical tokens token-level contrastsive estimation positive models negative models differential likelihood token-level DPO GSM8K MATH500 Llama-3 deepseek-math