title: BASS: Batched Attention-optimized Speculative Sampling
summary: Speculative decoding has emerged as a powerful method to improve latency and
  throughput in hosting large language models. However, most existing
  implementations focus on generating a single sequence. Real-world generative AI
  applications often require multiple responses and how to perform speculative
  decoding in a batched setting while preserving its latency benefits poses
  non-trivial challenges. This paper describes a system of batched speculative
  decoding that sets a new state of the art in multi-sequence generation latency
  and that demonstrates superior GPU utilization as well as quality of
  generations within a time budget. For example, for a 7.8B-size model on a
  single A100 GPU and with a batch size of 8, each sequence is generated at an
  average speed of 5.8ms per token, the overall throughput being 1.1K tokens per
  second. These results represent state-of-the-art latency and a 2.15X speed-up
  over optimized regular decoding. Within a time budget that regular decoding
  does not finish, our system is able to generate sequences with HumanEval
  Pass@First of 43% and Pass@All of 61%, far exceeding what's feasible with
  single-sequence speculative decoding. Our peak GPU utilization during decoding
  reaches as high as 15.8%, more than 3X the highest of that of regular decoding
  and around 10X of single-sequence speculative decoding.
paper_authors._id: 662a9330a57509c0c1493ef1
paper_authors.name: Haifeng Qian
paper_authors._id: 662a9330a57509c0c1493ef2
paper_authors.name: Sujan Kumar Gonugondla
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-25T17:30:25.419Z
paper_authors._id: 61787605267320abb99df76f
paper_authors.avatarUrl: /avatars/b78377529ddab0c23f67b7208b47dabf.svg
paper_authors.fullname: Gonugondla
paper_authors.type: user
paper_authors.user: SujanKumar
paper_authors._id: 662a9330a57509c0c1493ef3
paper_authors.name: Sungsoo Ha
paper_authors._id: 662a9330a57509c0c1493ef4
paper_authors.name: Mingyue Shang
paper_authors._id: 662a9330a57509c0c1493ef5
paper_authors.name: Sanjay Krishna Gouda
paper_authors._id: 662a9330a57509c0c1493ef6
paper_authors.name: Ramesh Nallapati
paper_authors._id: 662a9330a57509c0c1493ef7
paper_authors.name: Sudipta Sengupta
paper_authors._id: 662a9330a57509c0c1493ef8
paper_authors.name: Xiaofei Ma
paper_authors._id: 662a9330a57509c0c1493ef9
paper_authors.name: Anoop Deoras
paper_title: BASS: Batched Attention-optimized Speculative Sampling
paper_summary: Speculative decoding has emerged as a powerful method to improve latency and
  throughput in hosting large language models. However, most existing
  implementations focus on generating a single sequence. Real-world generative AI
  applications often require multiple responses and how to perform speculative
  decoding in a batched setting while preserving its latency benefits poses
  non-trivial challenges. This paper describes a system of batched speculative
  decoding that sets a new state of the art in multi-sequence generation latency
  and that demonstrates superior GPU utilization as well as quality of
  generations within a time budget. For example, for a 7.8B-size model on a
  single A100 GPU and with a batch size of 8, each sequence is generated at an
  average speed of 5.8ms per token, the overall throughput being 1.1K tokens per
  second. These results represent state-of-the-art latency and a 2.15X speed-up
  over optimized regular decoding. Within a time budget that regular decoding
  does not finish, our system is able to generate sequences with HumanEval
  Pass@First of 43% and Pass@All of 61%, far exceeding what's feasible with
  single-sequence speculative decoding. Our peak GPU utilization during decoding
  reaches as high as 15.8%, more than 3X the highest of that of regular decoding
  and around 10X of single-sequence speculative decoding.
paper_ai_summary: A system for batched speculative decoding enhances latency and throughput in generating multiple sequences from large language models, outperforming single-sequence speculative decoding in both speed and quality.
paper_ai_keywords: speculative decoding batched speculative decoding multi-sequence generation GPU utilization HumanEval Pass@First HumanEval Pass@All