title: A dynamic parallel method for performance optimization on hybrid CPUs
summary: The AIPC concept is gaining popularity, and more and more hybrid CPUs will be
  running AI models on client devices. However, the current AI inference
  framework overlooks the imbalanced hardware capability of hybrid CPUs, leading
  to low inference performance. To address this issue, we have introduced a
  dynamic parallel method for hybrid CPUs, which significantly increases LLM
  inference performance by balancing the workload for each core of a hybrid CPU
  before the parallel work starts. This method has enabled Neural Speed to
  achieve more than 90% (on average) of memory bandwidth on two hybrid Intel
  CPUs.
paper_authors._id: 674ffc79f3a3876a956ed67e
paper_authors.name: Luo Yu
paper_authors._id: 674ffc79f3a3876a956ed67f
paper_authors.name: Liu Yucheng
paper_authors._id: 674ffc79f3a3876a956ed680
paper_authors.name: Shen Haihao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:44:17.991Z
paper_authors._id: 60ac3318e3de7c7440abb850
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
paper_authors.fullname: Haihao Shen
paper_authors.type: user
paper_authors.user: Haihao
paper_title: A dynamic parallel method for performance optimization on hybrid CPUs
paper_summary: The AIPC concept is gaining popularity, and more and more hybrid CPUs will be
  running AI models on client devices. However, the current AI inference
  framework overlooks the imbalanced hardware capability of hybrid CPUs, leading
  to low inference performance. To address this issue, we have introduced a
  dynamic parallel method for hybrid CPUs, which significantly increases LLM
  inference performance by balancing the workload for each core of a hybrid CPU
  before the parallel work starts. This method has enabled Neural Speed to
  achieve more than 90% (on average) of memory bandwidth on two hybrid Intel
  CPUs.