title: Transformer-Lite: High-efficiency Deployment of Large Language Models on
    Mobile Phone GPUs
summary: The Large Language Model (LLM) is widely employed for tasks such as
  intelligent assistants, text summarization, translation, and multi-modality on
  mobile phones. However, the current methods for on-device LLM deployment
  maintain slow inference speed, which causes poor user experience. To facilitate
  high-efficiency LLM deployment on device GPUs, we propose four optimization
  techniques: (a) a symbolic expression-based approach to support dynamic shape
  model inference; (b) operator optimizations and execution priority setting to
  enhance inference speed and reduce phone lagging; (c) an FP4 quantization
  method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based
  technique to eliminate the need for copying KV cache after LLM inference.
  Furthermore, we implement these methods in our mobile inference engine,
  Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We
  evaluated Transformer-Lite's performance using LLMs with varied architectures
  and parameters ranging from 2B to 14B. Specifically, we achieved prefill and
  decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s
  and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based
  FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the
  prefill speed and 2~3x speedup for the decoding speed.
paper_authors._id: 660a421540e346fba5b97365
paper_authors.name: Luchang Li
paper_authors._id: 660a421540e346fba5b97366
paper_authors.name: Sheng Qian
paper_authors._id: 660a421540e346fba5b97367
paper_authors.name: Jie Lu
paper_authors._id: 660a421540e346fba5b97368
paper_authors.name: Lunxi Yuan
paper_authors._id: 660a421540e346fba5b97369
paper_authors.name: Rui Wang
paper_authors._id: 660a421540e346fba5b9736a
paper_authors.name: Qin Xie
paper_title: Transformer-Lite: High-efficiency Deployment of Large Language Models on
    Mobile Phone GPUs
paper_summary: The Large Language Model (LLM) is widely employed for tasks such as
  intelligent assistants, text summarization, translation, and multi-modality on
  mobile phones. However, the current methods for on-device LLM deployment
  maintain slow inference speed, which causes poor user experience. To facilitate
  high-efficiency LLM deployment on device GPUs, we propose four optimization
  techniques: (a) a symbolic expression-based approach to support dynamic shape
  model inference; (b) operator optimizations and execution priority setting to
  enhance inference speed and reduce phone lagging; (c) an FP4 quantization
  method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based
  technique to eliminate the need for copying KV cache after LLM inference.
  Furthermore, we implement these methods in our mobile inference engine,
  Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We
  evaluated Transformer-Lite's performance using LLMs with varied architectures
  and parameters ranging from 2B to 14B. Specifically, we achieved prefill and
  decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s
  and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based
  FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the
  prefill speed and 2~3x speedup for the decoding speed.
paper_ai_summary: Transformer-Lite optimizes LLM deployment on device GPUs through various techniques to achieve significant speedups in prefill and decoding over CPU and GPU-based methods.
paper_ai_keywords: symbolic expression-based dynamic shape model inference operator optimizations FP4 quantization M0E4 sub-tensor-based Transformer-Lite ChatGLM2 6B Gemma 2B prefill speed decoding speed FastLLM MLC-LLM