FlashDecoding++: Faster Large Language Model Inference on GPUs
As the Large Language Model (LLM) becomes increasingly important in various
domains. However, the following challenges still remain unsolved in
accelerating LLM inference: (1) Synchronized partial softmax update. The
softmax operation requires a synchronized update operation among each partial
softmax result, leading to ~20% overheads for the attention computation in
LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices
performing GEMM in LLM inference is flat, leading to under-utilized computation
and >50% performance loss after padding zeros in previous designs. (3)
Performance loss due to static dataflow. Kernel performance in LLM depends on
varied input data features, hardware configurations, etc. A single and static
dataflow may lead to a 50.25% performance loss for GEMMs of different shapes in
LLM inference.
  We present FlashDecoding++, a fast LLM inference engine supporting mainstream
LLMs and hardware back-ends. To tackle the above challenges, FlashDecoding++
creatively proposes: (1) Asynchronized softmax with unified max value.
FlashDecoding++ introduces a unified max value technique for different partial
softmax computations to avoid synchronization. (2) Flat GEMM optimization with
double buffering. FlashDecoding++ points out that flat GEMMs with different
shapes face varied bottlenecks. Then, techniques like double buffering are
introduced. (3) Heuristic dataflow with hardware resource adaptation.
FlashDecoding++ heuristically optimizes dataflow using different hardware
resource considering input dynamics. Due to the versatility of optimizations in
FlashDecoding++, FlashDecoding++ can achieve up to 4.86x and 2.18x speedup on
both NVIDIA and AMD GPUs compared to Hugging Face implementations.
FlashDecoding++ also achieves an average speedup of 1.37x compared to
state-of-the-art LLM inference engines on mainstream LLMs.
654503173353ea3d48939522
Ke Hong
admin_assigned
2023-11-03T15:55:08.064Z
6470a9394be5cf1f334a9db6
/avatars/feb19301d4a727f1395eb56fe73d3e54.svg
Ke Hong
user
hkeee
654503173353ea3d48939523
Guohao Dai
654503173353ea3d48939524
Jiaming Xu
admin_assigned
2023-11-03T15:54:34.313Z
651526b2391bec2898ba9103
/avatars/b965b5e5694141b79c4c0b7a98a66a07.svg
Jiaming Xu
user
MathsCode
654503173353ea3d48939525
Qiuli Mao
admin_assigned
2023-11-03T15:54:00.039Z
64be0b7978b89c4aa40c087d
/avatars/15b0986fd1f000965a1419a2c8b6e265.svg
Qiuli Mao
user
maoqiuli
654503173353ea3d48939526
Xiuhong Li
654503173353ea3d48939527
Jun Liu
654503173353ea3d48939528
Kangdi Chen
654503173353ea3d48939529
Hanyu Dong
admin_assigned
2023-11-03T15:53:05.692Z
641a9507f9dd6391a24607f5
/avatars/1319afb54e0c02d5dae046ddecbd9335.svg
Hanyudong
user
Hanyudong
654503173353ea3d4893952a
Yu Wang
FlashDecoding++: Faster Large Language Model Inference on GPUs
As the Large Language Model (LLM) becomes increasingly important in various
domains. However, the following challenges still remain unsolved in
accelerating LLM inference: (1) Synchronized partial softmax update. The
softmax operation requires a synchronized update operation among each partial
softmax result, leading to ~20% overheads for the attention computation in
LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices
performing GEMM in LLM inference is flat, leading to under-utilized computation
and >50% performance loss after padding zeros in previous designs. (3)
Performance loss due to static dataflow. Kernel performance in LLM depends on
varied input data features, hardware configurations, etc. A single and static
dataflow may lead to a 50.25% performance loss for GEMMs of different shapes in
LLM inference.
  We present FlashDecoding++, a fast LLM inference engine supporting mainstream
LLMs and hardware back-ends. To tackle the above challenges, FlashDecoding++
creatively proposes: (1) Asynchronized softmax with unified max value.
FlashDecoding++ introduces a unified max value technique for different partial
softmax computations to avoid synchronization. (2) Flat GEMM optimization with
double buffering. FlashDecoding++ points out that flat GEMMs with different
shapes face varied bottlenecks. Then, techniques like double buffering are
introduced. (3) Heuristic dataflow with hardware resource adaptation.
FlashDecoding++ heuristically optimizes dataflow using different hardware
resource considering input dynamics. Due to the versatility of optimizations in
FlashDecoding++, FlashDecoding++ can achieve up to 4.86x and 2.18x speedup on
both NVIDIA and AMD GPUs compared to Hugging Face implementations.
FlashDecoding++ also achieves an average speedup of 1.37x compared to
state-of-the-art LLM inference engines on mainstream LLMs.
FlashDecoding++ addresses challenges in LLM inference through asynchronized softmax, flat GEMM optimization, and heuristic dataflow adaptation, achieving significant speedups.
synchronized partial softmax update attention computation flat GEMM under-utilized computation static dataflow FlashDecoding++ asynchronized softmax unified max value flat GEMM optimization double buffering heuristic dataflow hardware resource adaptation