Inference Performance Optimization for Large Language Models on CPUs
Large language models (LLMs) have shown exceptional performance and vast
potential across diverse tasks. However, the deployment of LLMs with high
performance in low-resource environments has garnered significant attention in
the industry. When GPU hardware resources are limited, we can explore
alternative options on CPUs. To mitigate the financial burden and alleviate
constraints imposed by hardware resources, optimizing inference performance is
necessary. In this paper, we introduce an easily deployable inference
performance optimization solution aimed at accelerating LLMs on CPUs. In this
solution, we implement an effective way to reduce the KV cache size while
ensuring precision. We propose a distributed inference optimization approach
and implement it based on oneAPI Collective Communications Library.
Furthermore, we propose optimization approaches for LLMs on CPU, and conduct
tailored optimizations for the most commonly used models. The code is
open-sourced at https://github.com/intel/xFasterTransformer.
668f48815156d55f729b4724
Pujiang He
admin_assigned
2024-07-11T14:54:53.744Z
63b411487af2e415f25974f7
/avatars/dad07355462c1a7e296af237fb55a6fc.svg
pujiang
user
pujiang
668f48815156d55f729b4725
Shan Zhou
claimed_verified
2024-07-12T07:41:11.388Z
64004785b09f82a81a2672c1
/avatars/710f1570ec16b63a02c400842da341a5.svg
Shan Zhou
user
ShanShanZhou
668f48815156d55f729b4726
Wenhuan Huang
extracted_confirmed
2024-07-11T02:55:02.327Z
65e52cd1d99fd14f4c736353
/avatars/50abd36372e22f9a7c91ada6d7c08eff.svg
huang
user
wenhuanh
668f48815156d55f729b4727
Changqing Li
claimed_verified
2024-07-15T09:00:58.479Z
6564a58a3f025ef9f1107d2a
/avatars/375bc7229e937f94f838851140c5abe4.svg
Li Changqing
user
changqing2611
668f48815156d55f729b4728
Duyi Wang
admin_assigned
2024-07-11T10:21:01.964Z
65e16547f7d0ae1b62244f7b
/avatars/94053a9296500f44f7e933d091c0bc2c.svg
Duyi-Wang
user
Duyi-Wang
668f48815156d55f729b4729
Bin Guo
claimed_verified
2024-07-12T07:41:05.790Z
6672dd09c33b5004b68e5bff
/avatars/84bcf90b5b007a407ee57bb58f243b2e.svg
bin1guo
user
bin1guo
668f48815156d55f729b472a
Chen Meng
claimed_verified
2024-07-12T10:57:13.349Z
653648aeb66f4bf6893c43f8
/avatars/06d5f0401f90d5a81b215ba414b5028f.svg
Chen Meng
user
abenmao
668f48815156d55f729b472b
Sheng Gui
claimed_verified
2024-07-12T07:41:13.800Z
6690cb8561e8eed8d75baebe
/avatars/52bcc3c49f0107b871ac016d77c78e39.svg
sheng gui
user
sgui
668f48815156d55f729b472c
Weifei Yu
claimed_verified
2024-07-12T07:41:19.213Z
63f5743ab51da4d61daabf7c
/avatars/21c834fbeccef36b9712d908d0b734c2.svg
marvinYu
user
marvinYu
668f48815156d55f729b472d
Yi Xie
Inference Performance Optimization for Large Language Models on CPUs
Large language models (LLMs) have shown exceptional performance and vast
potential across diverse tasks. However, the deployment of LLMs with high
performance in low-resource environments has garnered significant attention in
the industry. When GPU hardware resources are limited, we can explore
alternative options on CPUs. To mitigate the financial burden and alleviate
constraints imposed by hardware resources, optimizing inference performance is
necessary. In this paper, we introduce an easily deployable inference
performance optimization solution aimed at accelerating LLMs on CPUs. In this
solution, we implement an effective way to reduce the KV cache size while
ensuring precision. We propose a distributed inference optimization approach
and implement it based on oneAPI Collective Communications Library.
Furthermore, we propose optimization approaches for LLMs on CPU, and conduct
tailored optimizations for the most commonly used models. The code is
open-sourced at https://github.com/intel/xFasterTransformer.
The paper presents a performance optimization solution for deploying large language models on CPUs by reducing KV cache size and implementing distributed inference, open-sourced for use.
large language models KV cache size distributed inference oneAPI Collective Communications Library