Yume: An Interactive World Generation Model
Yume aims to use images, text, or videos to create an interactive, realistic,
and dynamic world, which allows exploration and control using peripheral
devices or neural signals. In this report, we present a preview version of
\method, which creates a dynamic world from an input image and allows
exploration of the world using keyboard actions. To achieve this high-fidelity
and interactive video world generation, we introduce a well-designed framework,
which consists of four main components, including camera motion quantization,
video generation architecture, advanced sampler, and model acceleration. First,
we quantize camera motions for stable training and user-friendly interaction
using keyboard inputs. Then, we introduce the Masked Video Diffusion
Transformer~(MVDT) with a memory module for infinite video generation in an
autoregressive manner. After that, training-free Anti-Artifact Mechanism (AAM)
and Time Travel Sampling based on Stochastic Differential Equations (TTS-SDE)
are introduced to the sampler for better visual quality and more precise
control. Moreover, we investigate model acceleration by synergistic
optimization of adversarial distillation and caching mechanisms. We use the
high-quality world exploration dataset \sekai to train \method, and it achieves
remarkable results in diverse scenes and applications. All data, codebase, and
model weights are available on https://github.com/stdstu12/YUME. Yume will
update monthly to achieve its original goal. Project page:
https://stdstu12.github.io/YUME-Project/.
6881b7d1df7c5aafaf37f0d5
Xiaofeng Mao
6881b7d1df7c5aafaf37f0d6
Shaoheng Lin
6881b7d1df7c5aafaf37f0d7
Zhen Li
6881b7d1df7c5aafaf37f0d8
Chuanhao Li
6881b7d1df7c5aafaf37f0d9
Wenshuo Peng
6881b7d1df7c5aafaf37f0da
Tong He
6881b7d1df7c5aafaf37f0db
Jiangmiao Pang
admin_assigned
2025-07-24T12:00:40.580Z
65783ee6ee33d547aecc3ffc
https://cdn-avatars.huggingface.co/v1/production/uploads/65783ee6ee33d547aecc3ffc/lWZX88c-0dCsN-yB9Jhlf.jpeg
Jiangmiao Pang
user
Jiangmiao
6881b7d1df7c5aafaf37f0dc
Mingmin Chi
6881b7d1df7c5aafaf37f0dd
Yu Qiao
6881b7d1df7c5aafaf37f0de
Kaipeng Zhang
admin_assigned
2025-07-24T12:00:46.757Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
Yume: An Interactive World Generation Model
Yume aims to use images, text, or videos to create an interactive, realistic,
and dynamic world, which allows exploration and control using peripheral
devices or neural signals. In this report, we present a preview version of
\method, which creates a dynamic world from an input image and allows
exploration of the world using keyboard actions. To achieve this high-fidelity
and interactive video world generation, we introduce a well-designed framework,
which consists of four main components, including camera motion quantization,
video generation architecture, advanced sampler, and model acceleration. First,
we quantize camera motions for stable training and user-friendly interaction
using keyboard inputs. Then, we introduce the Masked Video Diffusion
Transformer~(MVDT) with a memory module for infinite video generation in an
autoregressive manner. After that, training-free Anti-Artifact Mechanism (AAM)
and Time Travel Sampling based on Stochastic Differential Equations (TTS-SDE)
are introduced to the sampler for better visual quality and more precise
control. Moreover, we investigate model acceleration by synergistic
optimization of adversarial distillation and caching mechanisms. We use the
high-quality world exploration dataset \sekai to train \method, and it achieves
remarkable results in diverse scenes and applications. All data, codebase, and
model weights are available on https://github.com/stdstu12/YUME. Yume will
update monthly to achieve its original goal. Project page:
https://stdstu12.github.io/YUME-Project/.
A framework for generating and exploring interactive, high-fidelity video worlds from images using a Masked Video Diffusion Transformer, advanced sampling techniques, and model acceleration.
camera motion quantization Masked Video Diffusion Transformer memory module autoregressive generation Anti-Artifact Mechanism Time Travel Sampling Stochastic Differential Equations adversarial distillation caching mechanisms