title: PockEngine: Sparse and Efficient Fine-tuning in a Pocket
summary: On-device learning and efficient fine-tuning enable continuous and
  privacy-preserving customization (e.g., locally fine-tuning large language
  models on personalized data). However, existing training frameworks are
  designed for cloud servers with powerful accelerators (e.g., GPUs, TPUs) and
  lack the optimizations for learning on the edge, which faces challenges of
  resource limitations and edge hardware diversity. We introduce PockEngine: a
  tiny, sparse and efficient engine to enable fine-tuning on various edge
  devices. PockEngine supports sparse backpropagation: it prunes the backward
  graph and sparsely updates the model with measured memory saving and latency
  reduction while maintaining the model quality. Secondly, PockEngine is
  compilation first: the entire training graph (including forward, backward and
  optimization steps) is derived at compile-time, which reduces the runtime
  overhead and brings opportunities for graph transformations. PockEngine also
  integrates a rich set of training graph optimizations, thus can further
  accelerate the training cost, including operator reordering and backend
  switching. PockEngine supports diverse applications, frontends and hardware
  backends: it flexibly compiles and tunes models defined in
  PyTorch/TensorFlow/Jax and deploys binaries to mobile CPU/GPU/DSPs. We
  evaluated PockEngine on both vision models and large language models.
  PockEngine achieves up to 15 times speedup over off-the-shelf TensorFlow
  (Raspberry Pi), 5.6 times memory saving back-propagation (Jetson AGX Orin).
  Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin
  at 550 tokens/s, 7.9times faster than the PyTorch.
paper_authors._id: 653f119c8a67c542ee1a755b
paper_authors.name: Ligeng Zhu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-30T05:22:40.065Z
paper_authors._id: 62b4b5beb25cb80fcf278354
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
paper_authors.fullname: Ligeng Zhu
paper_authors.type: user
paper_authors.user: Ligeng-Zhu
paper_authors._id: 653f119c8a67c542ee1a755c
paper_authors.name: Lanxiang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T08:51:37.425Z
paper_authors._id: 6301d6455e305a35cb0846a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6301d6455e305a35cb0846a7/aT2AtzRMSY_T3y02MIUap.jpeg
paper_authors.fullname: Lanxiang Hu
paper_authors.type: user
paper_authors.user: Snyhlxde
paper_authors._id: 653f119c8a67c542ee1a755d
paper_authors.name: Ji Lin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-30T02:59:29.559Z
paper_authors._id: 62f68e83d278a8f3e784de17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660325537221-62f68e83d278a8f3e784de17.jpeg
paper_authors.fullname: Ji Lin
paper_authors.type: user
paper_authors.user: jilinmit
paper_authors._id: 653f119c8a67c542ee1a755e
paper_authors.name: Wei-Chen Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-10-30T02:14:53.897Z
paper_authors._id: 6514d59cf95f39fd02acd9dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6514d59cf95f39fd02acd9dc/SSS28-Mjv2UdbGRoKdOZ7.jpeg
paper_authors.fullname: Wei-Chen Wang
paper_authors.type: user
paper_authors.user: wweichen
paper_authors._id: 653f119c8a67c542ee1a755f
paper_authors.name: Wei-Ming Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T14:13:20.147Z
paper_authors._id: 650215dd814e13d7fe99baef
paper_authors.avatarUrl: /avatars/20e1c93e82c8e44bd1258fabbc8ea427.svg
paper_authors.fullname: Wei-Ming Chen
paper_authors.type: user
paper_authors.user: meenchen
paper_authors._id: 653f119c8a67c542ee1a7560
paper_authors.name: Chuang Gan
paper_authors._id: 653f119c8a67c542ee1a7561
paper_authors.name: Song Han
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-10-30T02:14:53.897Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_title: PockEngine: Sparse and Efficient Fine-tuning in a Pocket
paper_summary: On-device learning and efficient fine-tuning enable continuous and
  privacy-preserving customization (e.g., locally fine-tuning large language
  models on personalized data). However, existing training frameworks are
  designed for cloud servers with powerful accelerators (e.g., GPUs, TPUs) and
  lack the optimizations for learning on the edge, which faces challenges of
  resource limitations and edge hardware diversity. We introduce PockEngine: a
  tiny, sparse and efficient engine to enable fine-tuning on various edge
  devices. PockEngine supports sparse backpropagation: it prunes the backward
  graph and sparsely updates the model with measured memory saving and latency
  reduction while maintaining the model quality. Secondly, PockEngine is
  compilation first: the entire training graph (including forward, backward and
  optimization steps) is derived at compile-time, which reduces the runtime
  overhead and brings opportunities for graph transformations. PockEngine also
  integrates a rich set of training graph optimizations, thus can further
  accelerate the training cost, including operator reordering and backend
  switching. PockEngine supports diverse applications, frontends and hardware
  backends: it flexibly compiles and tunes models defined in
  PyTorch/TensorFlow/Jax and deploys binaries to mobile CPU/GPU/DSPs. We
  evaluated PockEngine on both vision models and large language models.
  PockEngine achieves up to 15 times speedup over off-the-shelf TensorFlow
  (Raspberry Pi), 5.6 times memory saving back-propagation (Jetson AGX Orin).
  Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin
  at 550 tokens/s, 7.9times faster than the PyTorch.
paper_ai_summary: PockEngine is an efficient on-device learning engine that supports sparse backpropagation and graph optimizations for fine-tuning models on edge devices with varying hardware and constraints.
paper_ai_keywords: sparse backpropagation sparse update model quality compilation first training graph optimizations operator reordering backend switching on-device learning efficient fine-tuning edge devices edge hardware diversity mobile CPU mobile GPU DSPs PyTorch TensorFlow Jax vision models large language models LLaMav2-7B