title: FlashDecoding++: Faster Large Language Model Inference on GPUs
summary: As the Large Language Model (LLM) becomes increasingly important in various
  domains. However, the following challenges still remain unsolved in
  accelerating LLM inference: (1) Synchronized partial softmax update. The
  softmax operation requires a synchronized update operation among each partial
  softmax result, leading to ~20% overheads for the attention computation in
  LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices
  performing GEMM in LLM inference is flat, leading to under-utilized computation
  and >50% performance loss after padding zeros in previous designs. (3)
  Performance loss due to static dataflow. Kernel performance in LLM depends on
  varied input data features, hardware configurations, etc. A single and static
  dataflow may lead to a 50.25% performance loss for GEMMs of different shapes in
  LLM inference.
    We present FlashDecoding++, a fast LLM inference engine supporting mainstream
  LLMs and hardware back-ends. To tackle the above challenges, FlashDecoding++
  creatively proposes: (1) Asynchronized softmax with unified max value.
  FlashDecoding++ introduces a unified max value technique for different partial
  softmax computations to avoid synchronization. (2) Flat GEMM optimization with
  double buffering. FlashDecoding++ points out that flat GEMMs with different
  shapes face varied bottlenecks. Then, techniques like double buffering are
  introduced. (3) Heuristic dataflow with hardware resource adaptation.
  FlashDecoding++ heuristically optimizes dataflow using different hardware
  resource considering input dynamics. Due to the versatility of optimizations in
  FlashDecoding++, FlashDecoding++ can achieve up to 4.86x and 2.18x speedup on
  both NVIDIA and AMD GPUs compared to Hugging Face implementations.
  FlashDecoding++ also achieves an average speedup of 1.37x compared to
  state-of-the-art LLM inference engines on mainstream LLMs.
paper_authors._id: 654503173353ea3d48939522
paper_authors.name: Ke Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T15:55:08.064Z
paper_authors._id: 6470a9394be5cf1f334a9db6
paper_authors.avatarUrl: /avatars/feb19301d4a727f1395eb56fe73d3e54.svg
paper_authors.fullname: Ke Hong
paper_authors.type: user
paper_authors.user: hkeee
paper_authors._id: 654503173353ea3d48939523
paper_authors.name: Guohao Dai
paper_authors._id: 654503173353ea3d48939524
paper_authors.name: Jiaming Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T15:54:34.313Z
paper_authors._id: 651526b2391bec2898ba9103
paper_authors.avatarUrl: /avatars/b965b5e5694141b79c4c0b7a98a66a07.svg
paper_authors.fullname: Jiaming Xu
paper_authors.type: user
paper_authors.user: MathsCode
paper_authors._id: 654503173353ea3d48939525
paper_authors.name: Qiuli Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T15:54:00.039Z
paper_authors._id: 64be0b7978b89c4aa40c087d
paper_authors.avatarUrl: /avatars/15b0986fd1f000965a1419a2c8b6e265.svg
paper_authors.fullname: Qiuli Mao
paper_authors.type: user
paper_authors.user: maoqiuli
paper_authors._id: 654503173353ea3d48939526
paper_authors.name: Xiuhong Li
paper_authors._id: 654503173353ea3d48939527
paper_authors.name: Jun Liu
paper_authors._id: 654503173353ea3d48939528
paper_authors.name: Kangdi Chen
paper_authors._id: 654503173353ea3d48939529
paper_authors.name: Hanyu Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T15:53:05.692Z
paper_authors._id: 641a9507f9dd6391a24607f5
paper_authors.avatarUrl: /avatars/1319afb54e0c02d5dae046ddecbd9335.svg
paper_authors.fullname: Hanyudong
paper_authors.type: user
paper_authors.user: Hanyudong
paper_authors._id: 654503173353ea3d4893952a
paper_authors.name: Yu Wang
paper_title: FlashDecoding++: Faster Large Language Model Inference on GPUs
paper_summary: As the Large Language Model (LLM) becomes increasingly important in various
  domains. However, the following challenges still remain unsolved in
  accelerating LLM inference: (1) Synchronized partial softmax update. The
  softmax operation requires a synchronized update operation among each partial
  softmax result, leading to ~20% overheads for the attention computation in
  LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices
  performing GEMM in LLM inference is flat, leading to under-utilized computation
  and >50% performance loss after padding zeros in previous designs. (3)
  Performance loss due to static dataflow. Kernel performance in LLM depends on
  varied input data features, hardware configurations, etc. A single and static
  dataflow may lead to a 50.25% performance loss for GEMMs of different shapes in
  LLM inference.
    We present FlashDecoding++, a fast LLM inference engine supporting mainstream
  LLMs and hardware back-ends. To tackle the above challenges, FlashDecoding++
  creatively proposes: (1) Asynchronized softmax with unified max value.
  FlashDecoding++ introduces a unified max value technique for different partial
  softmax computations to avoid synchronization. (2) Flat GEMM optimization with
  double buffering. FlashDecoding++ points out that flat GEMMs with different
  shapes face varied bottlenecks. Then, techniques like double buffering are
  introduced. (3) Heuristic dataflow with hardware resource adaptation.
  FlashDecoding++ heuristically optimizes dataflow using different hardware
  resource considering input dynamics. Due to the versatility of optimizations in
  FlashDecoding++, FlashDecoding++ can achieve up to 4.86x and 2.18x speedup on
  both NVIDIA and AMD GPUs compared to Hugging Face implementations.
  FlashDecoding++ also achieves an average speedup of 1.37x compared to
  state-of-the-art LLM inference engines on mainstream LLMs.
paper_ai_keywords: synchronized partial softmax update asynchronized softmax unified max value flat GEMM double buffering heuristic dataflow hardware resource adaptation