title: MobileNMT: Enabling Translation in 15MB and 30ms
summary: Deploying NMT models on mobile devices is essential for privacy, low latency,
  and offline scenarios. For high model capacity, NMT models are rather large.
  Running these models on devices is challenging with limited storage, memory,
  computation, and power consumption. Existing work either only focuses on a
  single metric such as FLOPs or general engine which is not good at
  auto-regressive decoding. In this paper, we present MobileNMT, a system that
  can translate in 15MB and 30ms on devices. We propose a series of principles
  for model compression when combined with quantization. Further, we implement an
  engine that is friendly to INT8 and decoding. With the co-design of model and
  engine, compared with the existing system, we speed up 47.0x and save 99.5% of
  memory with only 11.6% loss of BLEU. The code is publicly available at
  https://github.com/zjersey/Lightseq-ARM.
paper_authors._id: 64814905856901b0edbda48c
paper_authors.name: Ye Lin
paper_authors._id: 64814905856901b0edbda48d
paper_authors.name: Xiaohui Wang
paper_authors._id: 64814905856901b0edbda48e
paper_authors.name: Zhexi Zhang
paper_authors._id: 64814905856901b0edbda48f
paper_authors.name: Mingxuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:49:23.570Z
paper_authors._id: 641f1cc26d51620635e1dd7b
paper_authors.avatarUrl: /avatars/58512d4fdaab900ea0bd5aace7d69e7c.svg
paper_authors.fullname: wangmingxuan
paper_authors.type: user
paper_authors.user: wmx6866
paper_authors._id: 64814905856901b0edbda490
paper_authors.name: Tong Xiao
paper_authors._id: 64814905856901b0edbda491
paper_authors.name: Jingbo Zhu
paper_title: MobileNMT: Enabling Translation in 15MB and 30ms
paper_summary: Deploying NMT models on mobile devices is essential for privacy, low latency,
  and offline scenarios. For high model capacity, NMT models are rather large.
  Running these models on devices is challenging with limited storage, memory,
  computation, and power consumption. Existing work either only focuses on a
  single metric such as FLOPs or general engine which is not good at
  auto-regressive decoding. In this paper, we present MobileNMT, a system that
  can translate in 15MB and 30ms on devices. We propose a series of principles
  for model compression when combined with quantization. Further, we implement an
  engine that is friendly to INT8 and decoding. With the co-design of model and
  engine, compared with the existing system, we speed up 47.0x and save 99.5% of
  memory with only 11.6% loss of BLEU. The code is publicly available at
  https://github.com/zjersey/Lightseq-ARM.
paper_ai_summary: MobileNMT system translates with low memory and fast decoding on mobile devices through model compression and INT8-friendly engine.
paper_ai_keywords: NMT models model compression quantization auto-regressive decoding INT8 BLEU