title: BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language
    Models
summary: Large language models (LLMs) commonly employ autoregressive generation during
  inference, leading to high memory bandwidth demand and consequently extended
  latency. To mitigate this inefficiency, we present Bi-directional Tuning for
  lossless Acceleration (BiTA), an innovative method expediting LLMs via
  streamlined semi-autoregressive generation and draft verification. Inspired by
  the concept of prompt tuning, we enhance LLMs with a parameter-efficient design
  called bi-directional tuning for the capability in semi-autoregressive
  generation. Employing efficient tree-based decoding, the models perform draft
  candidate generation and verification in parallel, ensuring outputs identical
  to their autoregressive counterparts under greedy sampling. BiTA serves as a
  lightweight plug-in module, seamlessly boosting the inference efficiency of
  existing LLMs without requiring additional assistance models or incurring
  significant extra memory costs. Applying the proposed BiTA, LLaMA-2-70B-Chat
  achieves a 2.7times speedup on the MT-Bench benchmark. Extensive experiments
  confirm our method surpasses state-of-the-art acceleration techniques.
paper_authors._id: 65b07cd7baebc9e3715336af
paper_authors.name: Feng Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-24T12:18:49.541Z
paper_authors._id: 6500215baf1e40214375ec7e
paper_authors.avatarUrl: /avatars/622f3e6db1915cfd3d35586f1f457cab.svg
paper_authors.fullname: Feng Lin
paper_authors.type: user
paper_authors.user: linfeng93
paper_authors._id: 65b07cd7baebc9e3715336b0
paper_authors.name: Hanling Yi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:15:01.806Z
paper_authors._id: 658b8d72e2db8a8e253f79db
paper_authors.avatarUrl: /avatars/36337ce7fa25626bc293b133e29d706a.svg
paper_authors.fullname: Hanling Yi
paper_authors.type: user
paper_authors.user: ANTMAN2
paper_authors._id: 65b07cd7baebc9e3715336b1
paper_authors.name: Hongbin Li
paper_authors._id: 65b07cd7baebc9e3715336b2
paper_authors.name: Yifan Yang
paper_authors._id: 65b07cd7baebc9e3715336b3
paper_authors.name: Xiaotian Yu
paper_authors._id: 65b07cd7baebc9e3715336b4
paper_authors.name: Guangming Lu
paper_authors._id: 65b07cd7baebc9e3715336b5
paper_authors.name: Rong Xiao
paper_title: BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language
    Models
paper_summary: Large language models (LLMs) commonly employ autoregressive generation during
  inference, leading to high memory bandwidth demand and consequently extended
  latency. To mitigate this inefficiency, we present Bi-directional Tuning for
  lossless Acceleration (BiTA), an innovative method expediting LLMs via
  streamlined semi-autoregressive generation and draft verification. Inspired by
  the concept of prompt tuning, we enhance LLMs with a parameter-efficient design
  called bi-directional tuning for the capability in semi-autoregressive
  generation. Employing efficient tree-based decoding, the models perform draft
  candidate generation and verification in parallel, ensuring outputs identical
  to their autoregressive counterparts under greedy sampling. BiTA serves as a
  lightweight plug-in module, seamlessly boosting the inference efficiency of
  existing LLMs without requiring additional assistance models or incurring
  significant extra memory costs. Applying the proposed BiTA, LLaMA-2-70B-Chat
  achieves a 2.7times speedup on the MT-Bench benchmark. Extensive experiments
  confirm our method surpasses state-of-the-art acceleration techniques.
paper_ai_summary: BiTA, a parameter-efficient bi-directional tuning method for LLMs, accelerates inference via parallel draft candidate generation and verification without additional memory costs.
paper_ai_keywords: autoregressive generation semi-autoregressive generation draft verification prompt tuning parameter-efficient tuning tree-based decoding MT-Bench benchmark