title: Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
summary: This work aims at decreasing the end-to-end generation latency of large
  language models (LLMs). One of the major causes of the high generation latency
  is the sequential decoding approach adopted by almost all state-of-the-art
  LLMs. In this work, motivated by the thinking and writing process of humans, we
  propose "Skeleton-of-Thought" (SoT), which guides LLMs to first generate the
  skeleton of the answer, and then conducts parallel API calls or batched
  decoding to complete the contents of each skeleton point in parallel. Not only
  does SoT provide considerable speed-up (up to 2.39x across 11 different LLMs),
  but it can also potentially improve the answer quality on several question
  categories in terms of diversity and relevance. SoT is an initial attempt at
  data-centric optimization for efficiency, and reveal the potential of pushing
  LLMs to think more like a human for answer quality.
paper_authors._id: 64c75426720d494bb2aa1b13
paper_authors.name: Xuefei Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-31T07:10:22.244Z
paper_authors._id: 641031b1a78453b8d96b8420
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
paper_authors.fullname: Xuefei Ning
paper_authors.type: user
paper_authors.user: Foxfi
paper_authors._id: 64c75426720d494bb2aa1b14
paper_authors.name: Zinan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:48.172Z
paper_authors._id: 64c832a8c547ed5243d29630
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c832a8c547ed5243d29630/a46V0xOVRVknM9mUjKkTf.jpeg
paper_authors.fullname: Zinan Lin
paper_authors.type: user
paper_authors.user: fjxmlzn
paper_authors._id: 64c75426720d494bb2aa1b15
paper_authors.name: Zixuan Zhou
paper_authors._id: 64c75426720d494bb2aa1b16
paper_authors.name: Huazhong Yang
paper_authors._id: 64c75426720d494bb2aa1b17
paper_authors.name: Yu Wang
paper_title: Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
paper_summary: This work aims at decreasing the end-to-end generation latency of large
  language models (LLMs). One of the major causes of the high generation latency
  is the sequential decoding approach adopted by almost all state-of-the-art
  LLMs. In this work, motivated by the thinking and writing process of humans, we
  propose "Skeleton-of-Thought" (SoT), which guides LLMs to first generate the
  skeleton of the answer, and then conducts parallel API calls or batched
  decoding to complete the contents of each skeleton point in parallel. Not only
  does SoT provide considerable speed-up (up to 2.39x across 11 different LLMs),
  but it can also potentially improve the answer quality on several question
  categories in terms of diversity and relevance. SoT is an initial attempt at
  data-centric optimization for efficiency, and reveal the potential of pushing
  LLMs to think more like a human for answer quality.
paper_ai_summary: "Skeleton-of-Thought" (SoT) method reduces generation latency and potentially improves answer quality by guiding LLMs to generate a skeleton first, followed by parallel content completion.
paper_ai_keywords: Skeleton-of-Thought LLMs sequential decoding parallel API calls batched decoding