title: On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for
    Mobile Devices
summary: We present On-device Sora, a first pioneering solution for diffusion-based
  on-device text-to-video generation that operates efficiently on
  smartphone-grade devices. Building on Open-Sora, On-device Sora applies three
  novel techniques to address the challenges of diffusion-based text-to-video
  generation on computation- and memory-limited mobile devices. First, Linear
  Proportional Leap (LPL) reduces the excessive denoising steps required in video
  diffusion through an efficient leap-based approach. Second, Temporal Dimension
  Token Merging (TDTM) minimizes intensive token-processing computation in
  attention layers by merging consecutive tokens along the temporal dimension.
  Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions
  large models into smaller blocks and loads them into memory for concurrent
  model inference, effectively addressing the challenges of limited device
  memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental
  evaluations demonstrate that it is capable of generating high-quality videos on
  the device, comparable to those produced by Open-Sora running on high-end GPUs.
  These results show that On-device Sora enables efficient and high-quality video
  generation on resource-constrained mobile devices, expanding accessibility,
  ensuring user privacy, reducing dependence on cloud infrastructure, and
  lowering associated costs. We envision the proposed On-device Sora as a
  significant first step toward democratizing state-of-the-art generative
  technologies, enabling video generation capabilities on commodity mobile and
  embedded devices. The code implementation is publicly available at an GitHub
  repository: https://github.com/eai-lab/On-device-Sora.
paper_authors._id: 67a98180d0dc1ed664297368
paper_authors.name: Bosung Kim
paper_authors._id: 67a98180d0dc1ed664297369
paper_authors.name: Kyuhwan Lee
paper_authors._id: 67a98180d0dc1ed66429736a
paper_authors.name: Isu Jeong
paper_authors._id: 67a98180d0dc1ed66429736b
paper_authors.name: Jungmin Cheon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:24:07.147Z
paper_authors._id: 65c9911ad3870f24084060f9
paper_authors.avatarUrl: /avatars/889ff75133203f9ed5b3c46cc67fb068.svg
paper_authors.fullname: Jungmin Cheon
paper_authors.type: user
paper_authors.user: Ruyan2
paper_authors._id: 67a98180d0dc1ed66429736c
paper_authors.name: Yeojin Lee
paper_authors._id: 67a98180d0dc1ed66429736d
paper_authors.name: Seulki Lee
paper_title: On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for
    Mobile Devices
paper_summary: We present On-device Sora, a first pioneering solution for diffusion-based
  on-device text-to-video generation that operates efficiently on
  smartphone-grade devices. Building on Open-Sora, On-device Sora applies three
  novel techniques to address the challenges of diffusion-based text-to-video
  generation on computation- and memory-limited mobile devices. First, Linear
  Proportional Leap (LPL) reduces the excessive denoising steps required in video
  diffusion through an efficient leap-based approach. Second, Temporal Dimension
  Token Merging (TDTM) minimizes intensive token-processing computation in
  attention layers by merging consecutive tokens along the temporal dimension.
  Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions
  large models into smaller blocks and loads them into memory for concurrent
  model inference, effectively addressing the challenges of limited device
  memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental
  evaluations demonstrate that it is capable of generating high-quality videos on
  the device, comparable to those produced by Open-Sora running on high-end GPUs.
  These results show that On-device Sora enables efficient and high-quality video
  generation on resource-constrained mobile devices, expanding accessibility,
  ensuring user privacy, reducing dependence on cloud infrastructure, and
  lowering associated costs. We envision the proposed On-device Sora as a
  significant first step toward democratizing state-of-the-art generative
  technologies, enabling video generation capabilities on commodity mobile and
  embedded devices. The code implementation is publicly available at an GitHub
  repository: https://github.com/eai-lab/On-device-Sora.
paper_ai_summary: On-device Sora efficiently performs diffusion-based text-to-video generation on smartphones using novel techniques to reduce computational and memory costs.
paper_ai_keywords: diffusion-based text-to-video generation smartphone-grade devices Open-Sora Linear Proportional Leap Temporal Dimension Token Merging Concurrent Inference with Dynamic Loading iPhone 15 Pro video diffusion attention layers model inference resource-constrained mobile devices