title: Inference Performance Optimization for Large Language Models on CPUs
summary: Large language models (LLMs) have shown exceptional performance and vast
  potential across diverse tasks. However, the deployment of LLMs with high
  performance in low-resource environments has garnered significant attention in
  the industry. When GPU hardware resources are limited, we can explore
  alternative options on CPUs. To mitigate the financial burden and alleviate
  constraints imposed by hardware resources, optimizing inference performance is
  necessary. In this paper, we introduce an easily deployable inference
  performance optimization solution aimed at accelerating LLMs on CPUs. In this
  solution, we implement an effective way to reduce the KV cache size while
  ensuring precision. We propose a distributed inference optimization approach
  and implement it based on oneAPI Collective Communications Library.
  Furthermore, we propose optimization approaches for LLMs on CPU, and conduct
  tailored optimizations for the most commonly used models. The code is
  open-sourced at https://github.com/intel/xFasterTransformer.
paper_authors._id: 668f48815156d55f729b4724
paper_authors.name: Pujiang He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T14:54:53.744Z
paper_authors._id: 63b411487af2e415f25974f7
paper_authors.avatarUrl: /avatars/dad07355462c1a7e296af237fb55a6fc.svg
paper_authors.fullname: pujiang
paper_authors.type: user
paper_authors.user: pujiang
paper_authors._id: 668f48815156d55f729b4725
paper_authors.name: Shan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:41:11.388Z
paper_authors._id: 64004785b09f82a81a2672c1
paper_authors.avatarUrl: /avatars/710f1570ec16b63a02c400842da341a5.svg
paper_authors.fullname: Shan Zhou
paper_authors.type: user
paper_authors.user: ShanShanZhou
paper_authors._id: 668f48815156d55f729b4726
paper_authors.name: Wenhuan Huang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-11T02:55:02.327Z
paper_authors._id: 65e52cd1d99fd14f4c736353
paper_authors.avatarUrl: /avatars/50abd36372e22f9a7c91ada6d7c08eff.svg
paper_authors.fullname: huang
paper_authors.type: user
paper_authors.user: wenhuanh
paper_authors._id: 668f48815156d55f729b4727
paper_authors.name: Changqing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:00:58.479Z
paper_authors._id: 6564a58a3f025ef9f1107d2a
paper_authors.avatarUrl: /avatars/375bc7229e937f94f838851140c5abe4.svg
paper_authors.fullname: Li Changqing
paper_authors.type: user
paper_authors.user: changqing2611
paper_authors._id: 668f48815156d55f729b4728
paper_authors.name: Duyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:21:01.964Z
paper_authors._id: 65e16547f7d0ae1b62244f7b
paper_authors.avatarUrl: /avatars/94053a9296500f44f7e933d091c0bc2c.svg
paper_authors.fullname: Duyi-Wang
paper_authors.type: user
paper_authors.user: Duyi-Wang
paper_authors._id: 668f48815156d55f729b4729
paper_authors.name: Bin Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:41:05.790Z
paper_authors._id: 6672dd09c33b5004b68e5bff
paper_authors.avatarUrl: /avatars/84bcf90b5b007a407ee57bb58f243b2e.svg
paper_authors.fullname: bin1guo
paper_authors.type: user
paper_authors.user: bin1guo
paper_authors._id: 668f48815156d55f729b472a
paper_authors.name: Chen Meng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T10:57:13.349Z
paper_authors._id: 653648aeb66f4bf6893c43f8
paper_authors.avatarUrl: /avatars/06d5f0401f90d5a81b215ba414b5028f.svg
paper_authors.fullname: Chen Meng
paper_authors.type: user
paper_authors.user: abenmao
paper_authors._id: 668f48815156d55f729b472b
paper_authors.name: Sheng Gui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:41:13.800Z
paper_authors._id: 6690cb8561e8eed8d75baebe
paper_authors.avatarUrl: /avatars/52bcc3c49f0107b871ac016d77c78e39.svg
paper_authors.fullname: sheng gui
paper_authors.type: user
paper_authors.user: sgui
paper_authors._id: 668f48815156d55f729b472c
paper_authors.name: Weifei Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:41:19.213Z
paper_authors._id: 63f5743ab51da4d61daabf7c
paper_authors.avatarUrl: /avatars/21c834fbeccef36b9712d908d0b734c2.svg
paper_authors.fullname: marvinYu
paper_authors.type: user
paper_authors.user: marvinYu
paper_authors._id: 668f48815156d55f729b472d
paper_authors.name: Yi Xie
paper_title: Inference Performance Optimization for Large Language Models on CPUs
paper_summary: Large language models (LLMs) have shown exceptional performance and vast
  potential across diverse tasks. However, the deployment of LLMs with high
  performance in low-resource environments has garnered significant attention in
  the industry. When GPU hardware resources are limited, we can explore
  alternative options on CPUs. To mitigate the financial burden and alleviate
  constraints imposed by hardware resources, optimizing inference performance is
  necessary. In this paper, we introduce an easily deployable inference
  performance optimization solution aimed at accelerating LLMs on CPUs. In this
  solution, we implement an effective way to reduce the KV cache size while
  ensuring precision. We propose a distributed inference optimization approach
  and implement it based on oneAPI Collective Communications Library.
  Furthermore, we propose optimization approaches for LLMs on CPU, and conduct
  tailored optimizations for the most commonly used models. The code is
  open-sourced at https://github.com/intel/xFasterTransformer.
paper_ai_keywords: KV cache size distributed inference optimization oneAPI Collective Communications Library