From Virtual Games to Real-World Play
We introduce RealPlay, a neural network-based real-world game engine that
enables interactive video generation from user control signals. Unlike prior
works focused on game-style visuals, RealPlay aims to produce photorealistic,
temporally consistent video sequences that resemble real-world footage. It
operates in an interactive loop: users observe a generated scene, issue a
control command, and receive a short video chunk in response. To enable such
realistic and responsive generation, we address key challenges including
iterative chunk-wise prediction for low-latency feedback, temporal consistency
across iterations, and accurate control response. RealPlay is trained on a
combination of labeled game data and unlabeled real-world videos, without
requiring real-world action annotations. Notably, we observe two forms of
generalization: (1) control transfer-RealPlay effectively maps control signals
from virtual to real-world scenarios; and (2) entity transfer-although training
labels originate solely from a car racing game, RealPlay generalizes to control
diverse real-world entities, including bicycles and pedestrians, beyond
vehicles. Project page can be found: https://wenqsun.github.io/RealPlay/
685a8f0e0e4ad7e21975872a
Wenqiang Sun
admin_assigned
2025-06-24T13:32:35.425Z
64897b1f0ec897cfe579a399
https://cdn-avatars.huggingface.co/v1/production/uploads/64897b1f0ec897cfe579a399/ICR_75b877BaSE94gjBuj.jpeg
wenq
user
wenqsun
685a8f0e0e4ad7e21975872b
Fangyun Wei
685a8f0e0e4ad7e21975872c
Jinjing Zhao
685a8f0e0e4ad7e21975872d
Xi Chen
685a8f0e0e4ad7e21975872e
Zilong Chen
admin_assigned
2025-06-24T13:33:00.269Z
644e3e5f030210812f413073
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uW8TKV2sds97lFBwnt6JK.jpeg
Zilong Chen
user
heheyas
685a8f0e0e4ad7e21975872f
Hongyang Zhang
685a8f0e0e4ad7e219758730
Jun Zhang
685a8f0e0e4ad7e219758731
Yan Lu
From Virtual Games to Real-World Play
We introduce RealPlay, a neural network-based real-world game engine that
enables interactive video generation from user control signals. Unlike prior
works focused on game-style visuals, RealPlay aims to produce photorealistic,
temporally consistent video sequences that resemble real-world footage. It
operates in an interactive loop: users observe a generated scene, issue a
control command, and receive a short video chunk in response. To enable such
realistic and responsive generation, we address key challenges including
iterative chunk-wise prediction for low-latency feedback, temporal consistency
across iterations, and accurate control response. RealPlay is trained on a
combination of labeled game data and unlabeled real-world videos, without
requiring real-world action annotations. Notably, we observe two forms of
generalization: (1) control transfer-RealPlay effectively maps control signals
from virtual to real-world scenarios; and (2) entity transfer-although training
labels originate solely from a car racing game, RealPlay generalizes to control
diverse real-world entities, including bicycles and pedestrians, beyond
vehicles. Project page can be found: https://wenqsun.github.io/RealPlay/
RealPlay generates photorealistic, temporally consistent video sequences from user control signals through iterative prediction and generalizes to various real-world entities.
neural network-based real-world game engine interactive video generation control signals photorealistic temporally consistent video sequences chunk-wise prediction low-latency feedback temporal consistency control response labeled game data unlabeled real-world videos control transfer entity transfer car racing game bicycles pedestrians