Diffusion Models Are Real-Time Game Engines
Dani Valevski Yaniv Leviathan Moab Arar Shlomi Fruchter
We present GameNGen, the first game engine powered entirely by a neural model that also enables real-time interaction with a complex environment over long trajectories at high quality. When trained on the classic game DOOM, GameNGen extracts gameplay and uses it to generate a playable environment that can interactively simulate new trajectories. GameNGen runs at 20 frames per second on a single TPU and remains stable over extended multi-minute play sessions. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation, even after 5 minutes of auto-regressive generation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations help ensure stable auto-regressive generation over long trajectories, and decoder fine-tuning improves the fidelity of visual details and text.
Diffusion Models, Neural Game Engines, Real-Time Simulation, Reinforcement Learning, Auto-Regressive Generation, Video Game Simulation, Interactive World Simulation
GameNGen uses diffusion model and reinforcement learning to simulate a complex video game (DOOM) with high visual fidelity and stable gameplay.