title: Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads
    to Answers Faster
summary: In this work, we propose FastCoT, a model-agnostic framework based on
  parallel decoding without any further training of an auxiliary model or
  modification to the LLM itself. FastCoT uses a size-varying context window
  whose size changes with position to conduct parallel decoding and
  auto-regressive decoding simultaneously, thus fully utilizing GPU computation
  resources. In FastCoT, the parallel decoding part provides the LLM with a quick
  glance of the future composed of approximate tokens, which could lead to faster
  answers compared to regular autoregressive decoding used by causal
  transformers. We also provide an implementation of parallel decoding within
  LLM, which supports KV-cache generation and batch processing. Through extensive
  experiments, we demonstrate that FastCoT saves inference time by nearly 20%
  with only a negligible performance drop compared to the regular approach.
  Additionally, we show that the context window size exhibits considerable
  robustness for different tasks.
paper_authors._id: 655446884045e53f450414d3
paper_authors.name: Hongxuan Zhang
paper_authors._id: 655446884045e53f450414d4
paper_authors.name: Zhining Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:02:37.504Z
paper_authors._id: 61362d107c6d9b8e41469bda
paper_authors.avatarUrl: /avatars/b0588156bb9da87f05ed89dc9cf0a378.svg
paper_authors.fullname: Zhining Liu
paper_authors.type: user
paper_authors.user: Zhin
paper_authors._id: 655446884045e53f450414d5
paper_authors.name: Jiaqi Zheng
paper_authors._id: 655446884045e53f450414d6
paper_authors.name: Chenyi Zhuang
paper_authors._id: 655446884045e53f450414d7
paper_authors.name: Jinjie Gu
paper_authors._id: 655446884045e53f450414d8
paper_authors.name: Guihai Chen
paper_title: Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads
    to Answers Faster
paper_summary: In this work, we propose FastCoT, a model-agnostic framework based on
  parallel decoding without any further training of an auxiliary model or
  modification to the LLM itself. FastCoT uses a size-varying context window
  whose size changes with position to conduct parallel decoding and
  auto-regressive decoding simultaneously, thus fully utilizing GPU computation
  resources. In FastCoT, the parallel decoding part provides the LLM with a quick
  glance of the future composed of approximate tokens, which could lead to faster
  answers compared to regular autoregressive decoding used by causal
  transformers. We also provide an implementation of parallel decoding within
  LLM, which supports KV-cache generation and batch processing. Through extensive
  experiments, we demonstrate that FastCoT saves inference time by nearly 20%
  with only a negligible performance drop compared to the regular approach.
  Additionally, we show that the context window size exhibits considerable
  robustness for different tasks.
paper_ai_summary: FastCoT, a model-agnostic framework, leverages parallel decoding with a dynamic context window to reduce inference time for large language models without significant performance degradation.
paper_ai_keywords: FastCoT parallel decoding context window LLM auto-regressive decoding causal transformers KV-cache batch processing