title: An Efficient Sparse Inference Software Accelerator for Transformer-based
    Language Models on CPUs
summary: In recent years, Transformer-based language models have become the standard
  approach for natural language processing tasks. However, stringent throughput
  and latency requirements in industrial applications are limiting their
  adoption. To mitigate the gap, model compression techniques such as structured
  pruning are being used to improve inference efficiency. However, most existing
  neural network inference runtimes lack adequate support for structured
  sparsity. In this paper, we propose an efficient sparse deep learning inference
  software stack for Transformer-based language models where the weights are
  pruned with constant block size. Our sparse software accelerator leverages
  Intel Deep Learning Boost to maximize the performance of sparse matrix - dense
  matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel
  outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an
  order of magnitude on a wide range of GEMM shapes under 5 representative
  sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up
  to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library
  widely used in industry. We apply our sparse accelerator on widely-used
  Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base,
  and BERT-Large. Our sparse inference software shows up to 1.5x speedup over
  Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web
  Services under proxy production latency constraints. We also compare our
  solution with two framework-based inference solutions, ONNX Runtime and
  PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over
  PyTorch on Xeon under the latency constraints. All the source code is publicly
  available on Github: https://github.com/intel/intel-extension-for-transformers.
paper_authors._id: 649e3821ed55df9fd24331b2
paper_authors.name: Haihao Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T20:17:58.616Z
paper_authors._id: 60ac3318e3de7c7440abb850
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
paper_authors.fullname: Haihao Shen
paper_authors.type: user
paper_authors.user: Haihao
paper_authors._id: 649e3821ed55df9fd24331b3
paper_authors.name: Hengyu Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T20:18:13.773Z
paper_authors._id: 63edfe1c5f1300034dd90a37
paper_authors.avatarUrl: /avatars/fa742c6da029ce73b395d1a12a97753f.svg
paper_authors.fullname: Hyuno
paper_authors.type: user
paper_authors.user: HengyuMeng
paper_authors._id: 649e3821ed55df9fd24331b4
paper_authors.name: Bo Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T20:18:30.023Z
paper_authors._id: 616e6906a6e9a95e08a09c9d
paper_authors.avatarUrl: /avatars/2a8dece24cbc1779eb906d5582573c49.svg
paper_authors.fullname: Dong, Bo
paper_authors.type: user
paper_authors.user: BoDong
paper_authors._id: 649e3821ed55df9fd24331b5
paper_authors.name: Zhe Wang
paper_authors._id: 649e3821ed55df9fd24331b6
paper_authors.name: Ofir Zafrir
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T20:40:15.668Z
paper_authors._id: 5f8907c65d083370c711f284
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616423186722-5f8907c65d083370c711f284.jpeg
paper_authors.fullname: Ofir Zafrir
paper_authors.type: user
paper_authors.user: ofirzaf
paper_authors._id: 649e3821ed55df9fd24331b7
paper_authors.name: Yi Ding
paper_authors._id: 649e3821ed55df9fd24331b8
paper_authors.name: Yu Luo
paper_authors._id: 649e3821ed55df9fd24331b9
paper_authors.name: Hanwen Chang
paper_authors._id: 649e3821ed55df9fd24331ba
paper_authors.name: Qun Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T20:42:24.684Z
paper_authors._id: 63079d748b054d85f299e1e4
paper_authors.avatarUrl: /avatars/d4947439475dc81f2c9e9304382b6257.svg
paper_authors.fullname: Qun Gao
paper_authors.type: user
paper_authors.user: qgao007
paper_authors._id: 649e3821ed55df9fd24331bb
paper_authors.name: Ziheng Wang
paper_authors._id: 649e3821ed55df9fd24331bc
paper_authors.name: Guy Boudoukh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T20:42:55.307Z
paper_authors._id: 60abe77d08805574fcf076f2
paper_authors.avatarUrl: /avatars/06df4ead5a2014480c128103b9862f98.svg
paper_authors.fullname: Guy Boudoukh
paper_authors.type: user
paper_authors.user: guybd
paper_authors._id: 649e3821ed55df9fd24331bd
paper_authors.name: Moshe Wasserblat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T20:43:12.249Z
paper_authors._id: 6013c64031d84a3bb7650038
paper_authors.avatarUrl: /avatars/e05ca715004b39e79472399f75010bda.svg
paper_authors.fullname: Moshe Wasserblat
paper_authors.type: user
paper_authors.user: moshew
paper_title: An Efficient Sparse Inference Software Accelerator for Transformer-based
    Language Models on CPUs
paper_summary: In recent years, Transformer-based language models have become the standard
  approach for natural language processing tasks. However, stringent throughput
  and latency requirements in industrial applications are limiting their
  adoption. To mitigate the gap, model compression techniques such as structured
  pruning are being used to improve inference efficiency. However, most existing
  neural network inference runtimes lack adequate support for structured
  sparsity. In this paper, we propose an efficient sparse deep learning inference
  software stack for Transformer-based language models where the weights are
  pruned with constant block size. Our sparse software accelerator leverages
  Intel Deep Learning Boost to maximize the performance of sparse matrix - dense
  matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel
  outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an
  order of magnitude on a wide range of GEMM shapes under 5 representative
  sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up
  to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library
  widely used in industry. We apply our sparse accelerator on widely-used
  Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base,
  and BERT-Large. Our sparse inference software shows up to 1.5x speedup over
  Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web
  Services under proxy production latency constraints. We also compare our
  solution with two framework-based inference solutions, ONNX Runtime and
  PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over
  PyTorch on Xeon under the latency constraints. All the source code is publicly
  available on Github: https://github.com/intel/intel-extension-for-transformers.
paper_ai_keywords: Transformer-based language models weight pruning structured sparsity Intel Deep Learning Boost sparse matrix - dense matrix multiplication (SpMM) confidence interval GEMM sparsity ratio Bert-Mini DistilBERT Bert-Base BERT-Large Neural Magic's Deepsparse ONNX Runtime PyTorch Xeon Amazon Web Services proxy production latency constraints