title: Diffusion Models Are Real-Time Game Engines
summary: We present GameNGen, the first game engine powered entirely by a neural model
  that enables real-time interaction with a complex environment over long
  trajectories at high quality. GameNGen can interactively simulate the classic
  game DOOM at over 20 frames per second on a single TPU. Next frame prediction
  achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are
  only slightly better than random chance at distinguishing short clips of the
  game from clips of the simulation. GameNGen is trained in two phases: (1) an
  RL-agent learns to play the game and the training sessions are recorded, and
  (2) a diffusion model is trained to produce the next frame, conditioned on the
  sequence of past frames and actions. Conditioning augmentations enable stable
  auto-regressive generation over long trajectories.
paper_authors._id: 66ce94d1d3d85430b7ce06ad
paper_authors.name: Dani Valevski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:03:06.126Z
paper_authors._id: 632b5adf6110e37dba3e3ca3
paper_authors.avatarUrl: /avatars/2c59a399d9b5d01d116fcb7775761fc8.svg
paper_authors.fullname: Dani Valevski
paper_authors.type: user
paper_authors.user: daniva
paper_authors._id: 66ce94d1d3d85430b7ce06ae
paper_authors.name: Yaniv Leviathan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:03:12.296Z
paper_authors._id: 6394a2244db1a9bedd1f2229
paper_authors.avatarUrl: /avatars/f04496811eb6873781e2ba606a2f97f0.svg
paper_authors.fullname: Yaniv Leviathan
paper_authors.type: user
paper_authors.user: yanivle
paper_authors._id: 66ce94d1d3d85430b7ce06af
paper_authors.name: Moab Arar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T07:50:53.649Z
paper_authors._id: 60defb17bb3e1aaa18b005bb
paper_authors.avatarUrl: /avatars/086264e93d6f7ba2eab5cc1b218df4a5.svg
paper_authors.fullname: Moab Arar
paper_authors.type: user
paper_authors.user: moabarar
paper_authors._id: 66ce94d1d3d85430b7ce06b0
paper_authors.name: Shlomi Fruchter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:03:17.576Z
paper_authors._id: 6335ae887fcbeb615b0f9924
paper_authors.avatarUrl: /avatars/60fd9305ccc56f17d6007f781c54fe2b.svg
paper_authors.fullname: Shlomi Fruchter
paper_authors.type: user
paper_authors.user: shlomif
paper_title: Diffusion Models Are Real-Time Game Engines
paper_summary: We present GameNGen, the first game engine powered entirely by a neural model
  that enables real-time interaction with a complex environment over long
  trajectories at high quality. GameNGen can interactively simulate the classic
  game DOOM at over 20 frames per second on a single TPU. Next frame prediction
  achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are
  only slightly better than random chance at distinguishing short clips of the
  game from clips of the simulation. GameNGen is trained in two phases: (1) an
  RL-agent learns to play the game and the training sessions are recorded, and
  (2) a diffusion model is trained to produce the next frame, conditioned on the
  sequence of past frames and actions. Conditioning augmentations enable stable
  auto-regressive generation over long trajectories.
paper_ai_keywords: neural model real-time interaction next frame prediction PSNR lossy JPEG compression RL-agent diffusion model auto-regressive generation