Diffusion for World Modeling: Visual Details Matter in Atari
World models constitute a promising approach for training reinforcement
learning agents in a safe and sample-efficient manner. Recent world models
predominantly operate on sequences of discrete latent variables to model
environment dynamics. However, this compression into a compact discrete
representation may ignore visual details that are important for reinforcement
learning. Concurrently, diffusion models have become a dominant approach for
image generation, challenging well-established methods modeling discrete
latents. Motivated by this paradigm shift, we introduce DIAMOND (DIffusion As a
Model Of eNvironment Dreams), a reinforcement learning agent trained in a
diffusion world model. We analyze the key design choices that are required to
make diffusion suitable for world modeling, and demonstrate how improved visual
details can lead to improved agent performance. DIAMOND achieves a mean human
normalized score of 1.46 on the competitive Atari 100k benchmark; a new best
for agents trained entirely within a world model. To foster future research on
diffusion for world modeling, we release our code, agents and playable world
models at https://github.com/eloialonso/diamond.
664d594b9299555621c06496
Eloi Alonso
admin_assigned
2024-05-22T09:12:34.680Z
66476d39156169f872095c0c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ja4hmXYNBcprZau6Miux1.png
Eloi Alonso
user
eloialonso
664d594b9299555621c06497
Adam Jelley
admin_assigned
2024-05-22T09:12:52.406Z
66476cf764454e8f1c792345
https://cdn-avatars.huggingface.co/v1/production/uploads/66476cf764454e8f1c792345/MLviMsGieflLURaFxGbUP.jpeg
Adam Jelley
user
AdamJelley
664d594b9299555621c06498
Vincent Micheli
admin_assigned
2024-05-22T09:13:28.538Z
5f0d599719cb630495b81524
/avatars/3c7472d1644d0c78331fafa88d16f0a6.svg
Vincent Micheli
user
vincem
664d594b9299555621c06499
Anssi Kanervisto
664d594b9299555621c0649a
Amos Storkey
664d594b9299555621c0649b
Tim Pearce
admin_assigned
2024-05-22T09:13:41.205Z
636f5c246cd69d9a36ff8791
https://cdn-avatars.huggingface.co/v1/production/uploads/636f5c246cd69d9a36ff8791/CkqAKquinHhFsVu7qxBJm.jpeg
Tim Pearce
user
TeaPearce
664d594b9299555621c0649c
François Fleuret
admin_assigned
2024-05-22T09:13:52.895Z
6318c37e69ef5467e18fc63f
/avatars/7f1ac593608a89926f4107b4c3dd91d4.svg
François Fleuret
user
francoisfleuret
Diffusion for World Modeling: Visual Details Matter in Atari
World models constitute a promising approach for training reinforcement
learning agents in a safe and sample-efficient manner. Recent world models
predominantly operate on sequences of discrete latent variables to model
environment dynamics. However, this compression into a compact discrete
representation may ignore visual details that are important for reinforcement
learning. Concurrently, diffusion models have become a dominant approach for
image generation, challenging well-established methods modeling discrete
latents. Motivated by this paradigm shift, we introduce DIAMOND (DIffusion As a
Model Of eNvironment Dreams), a reinforcement learning agent trained in a
diffusion world model. We analyze the key design choices that are required to
make diffusion suitable for world modeling, and demonstrate how improved visual
details can lead to improved agent performance. DIAMOND achieves a mean human
normalized score of 1.46 on the competitive Atari 100k benchmark; a new best
for agents trained entirely within a world model. To foster future research on
diffusion for world modeling, we release our code, agents and playable world
models at https://github.com/eloialonso/diamond.
DIAMOND, a reinforcement learning agent using a diffusion world model, achieves top performance on Atari 100k by capturing visual details effectively.
world models reinforcement learning discrete latent variables diffusion models Atari 100k benchmark mean human normalized score