On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for
  Mobile Devices
We present On-device Sora, a first pioneering solution for diffusion-based
on-device text-to-video generation that operates efficiently on
smartphone-grade devices. Building on Open-Sora, On-device Sora applies three
novel techniques to address the challenges of diffusion-based text-to-video
generation on computation- and memory-limited mobile devices. First, Linear
Proportional Leap (LPL) reduces the excessive denoising steps required in video
diffusion through an efficient leap-based approach. Second, Temporal Dimension
Token Merging (TDTM) minimizes intensive token-processing computation in
attention layers by merging consecutive tokens along the temporal dimension.
Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions
large models into smaller blocks and loads them into memory for concurrent
model inference, effectively addressing the challenges of limited device
memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental
evaluations demonstrate that it is capable of generating high-quality videos on
the device, comparable to those produced by Open-Sora running on high-end GPUs.
These results show that On-device Sora enables efficient and high-quality video
generation on resource-constrained mobile devices, expanding accessibility,
ensuring user privacy, reducing dependence on cloud infrastructure, and
lowering associated costs. We envision the proposed On-device Sora as a
significant first step toward democratizing state-of-the-art generative
technologies, enabling video generation capabilities on commodity mobile and
embedded devices. The code implementation is publicly available at an GitHub
repository: https://github.com/eai-lab/On-device-Sora.
67a98180d0dc1ed664297368
Bosung Kim
67a98180d0dc1ed664297369
Kyuhwan Lee
67a98180d0dc1ed66429736a
Isu Jeong
67a98180d0dc1ed66429736b
Jungmin Cheon
admin_assigned
2025-02-10T16:24:07.147Z
65c9911ad3870f24084060f9
/avatars/889ff75133203f9ed5b3c46cc67fb068.svg
Jungmin Cheon
user
Ruyan2
67a98180d0dc1ed66429736c
Yeojin Lee
67a98180d0dc1ed66429736d
Seulki Lee
On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for
  Mobile Devices
We present On-device Sora, a first pioneering solution for diffusion-based
on-device text-to-video generation that operates efficiently on
smartphone-grade devices. Building on Open-Sora, On-device Sora applies three
novel techniques to address the challenges of diffusion-based text-to-video
generation on computation- and memory-limited mobile devices. First, Linear
Proportional Leap (LPL) reduces the excessive denoising steps required in video
diffusion through an efficient leap-based approach. Second, Temporal Dimension
Token Merging (TDTM) minimizes intensive token-processing computation in
attention layers by merging consecutive tokens along the temporal dimension.
Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions
large models into smaller blocks and loads them into memory for concurrent
model inference, effectively addressing the challenges of limited device
memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental
evaluations demonstrate that it is capable of generating high-quality videos on
the device, comparable to those produced by Open-Sora running on high-end GPUs.
These results show that On-device Sora enables efficient and high-quality video
generation on resource-constrained mobile devices, expanding accessibility,
ensuring user privacy, reducing dependence on cloud infrastructure, and
lowering associated costs. We envision the proposed On-device Sora as a
significant first step toward democratizing state-of-the-art generative
technologies, enabling video generation capabilities on commodity mobile and
embedded devices. The code implementation is publicly available at an GitHub
repository: https://github.com/eai-lab/On-device-Sora.
On-device Sora efficiently performs diffusion-based text-to-video generation on smartphones using novel techniques to reduce computational and memory costs.
diffusion-based text-to-video generation smartphone-grade devices Open-Sora Linear Proportional Leap Temporal Dimension Token Merging Concurrent Inference with Dynamic Loading iPhone 15 Pro video diffusion attention layers model inference resource-constrained mobile devices