title: Efficient LLM inference solution on Intel GPU
summary: Transformer based Large Language Models (LLMs) have been widely used in many
  fields, and the efficiency of LLM inference becomes hot topic in real
  applications. However, LLMs are usually complicatedly designed in model
  structure with massive operations and perform inference in the auto-regressive
  mode, making it a challenging task to design a system with high efficiency.
    In this paper, we propose an efficient LLM inference solution with low
  latency and high throughput. Firstly, we simplify the LLM decoder layer by
  fusing data movement and element-wise operations to reduce the memory access
  frequency and lower system latency. We also propose a segment KV cache policy
  to keep key/value of the request and response tokens in separate physical
  memory for effective device memory management, helping enlarge the runtime
  batch size and improve system throughput. A customized
  Scaled-Dot-Product-Attention kernel is designed to match our fusion policy
  based on the segment KV cache solution. We implement our LLM inference solution
  on Intel GPU and publish it publicly. Compared with the standard HuggingFace
  implementation, the proposed solution achieves up to 7x lower token latency and
  27x higher throughput for some popular LLMs on Intel GPU.
paper_authors._id: 65a09e24e0b49a84d21cca89
paper_authors.name: Hui Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-12T10:46:08.306Z
paper_authors._id: 65a1185de96941538121aa81
paper_authors.avatarUrl: /avatars/27c5947d20cc91bf5ef61b48d2c50a9c.svg
paper_authors.fullname: Hui Wu
paper_authors.type: user
paper_authors.user: wuhuikxCSU
paper_authors._id: 65a09e24e0b49a84d21cca8a
paper_authors.name: Yi Gan
paper_authors._id: 65a09e24e0b49a84d21cca8b
paper_authors.name: Feng Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-15T07:42:04.025Z
paper_authors._id: 65a4823541b6ef119c5a13a4
paper_authors.avatarUrl: /avatars/d57e51c4fa0bd3da1d1a96e1a18050f2.svg
paper_authors.fullname: Feng Yuan
paper_authors.type: user
paper_authors.user: arthuryuan1987
paper_authors._id: 65a09e24e0b49a84d21cca8c
paper_authors.name: Jing Ma
paper_authors._id: 65a09e24e0b49a84d21cca8d
paper_authors.name: Wei Zhu
paper_authors._id: 65a09e24e0b49a84d21cca8e
paper_authors.name: Yutao Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:39:14.660Z
paper_authors._id: 642b94595bf2355d02a3c2d7
paper_authors.avatarUrl: /avatars/23db8b19705a5e5f0b2bc2e2f4b0d8d0.svg
paper_authors.fullname: xuyutao
paper_authors.type: user
paper_authors.user: vqaFW
paper_authors._id: 65a09e24e0b49a84d21cca8f
paper_authors.name: Hong Zhu
paper_authors._id: 65a09e24e0b49a84d21cca90
paper_authors.name: Yuhua Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:39:29.465Z
paper_authors._id: 6481332d8c6a3b8f11fc40c7
paper_authors.avatarUrl: /avatars/16aee3d0e7ca35cf0d036aa71e7738e2.svg
paper_authors.fullname: zhuyuhua
paper_authors.type: user
paper_authors.user: TakaRika
paper_authors._id: 65a09e24e0b49a84d21cca91
paper_authors.name: Xiaoli Liu
paper_authors._id: 65a09e24e0b49a84d21cca92
paper_authors.name: Jinghui Gu
paper_title: Efficient LLM inference solution on Intel GPU
paper_summary: Transformer based Large Language Models (LLMs) have been widely used in many
  fields, and the efficiency of LLM inference becomes hot topic in real
  applications. However, LLMs are usually complicatedly designed in model
  structure with massive operations and perform inference in the auto-regressive
  mode, making it a challenging task to design a system with high efficiency.
    In this paper, we propose an efficient LLM inference solution with low
  latency and high throughput. Firstly, we simplify the LLM decoder layer by
  fusing data movement and element-wise operations to reduce the memory access
  frequency and lower system latency. We also propose a segment KV cache policy
  to keep key/value of the request and response tokens in separate physical
  memory for effective device memory management, helping enlarge the runtime
  batch size and improve system throughput. A customized
  Scaled-Dot-Product-Attention kernel is designed to match our fusion policy
  based on the segment KV cache solution. We implement our LLM inference solution
  on Intel GPU and publish it publicly. Compared with the standard HuggingFace
  implementation, the proposed solution achieves up to 7x lower token latency and
  27x higher throughput for some popular LLMs on Intel GPU.
paper_ai_summary: The proposed LLM inference solution reduces latency and increases throughput by simplifying decoder layers, using a segment KV cache policy, and optimizing the Scaled-Dot-Product-Attention kernel.
paper_ai_keywords: Transformer Large Language Models LLM inference auto-regressive memory access segment KV cache Scaled-Dot-Product-Attention Intel GPU