Efficient World Models with Context-Aware Tokenization
Scaling up deep Reinforcement Learning (RL) methods presents a significant
challenge. Following developments in generative modelling, model-based RL
positions itself as a strong contender. Recent advances in sequence modelling
have led to effective transformer-based world models, albeit at the price of
heavy computations due to the long sequences of tokens required to accurately
simulate environments. In this work, we propose Delta-IRIS, a new agent with
a world model architecture composed of a discrete autoencoder that encodes
stochastic deltas between time steps and an autoregressive transformer that
predicts future deltas by summarizing the current state of the world with
continuous tokens. In the Crafter benchmark, Delta-IRIS sets a new state of
the art at multiple frame budgets, while being an order of magnitude faster to
train than previous attention-based approaches. We release our code and models
at https://github.com/vmicheli/delta-iris.
66825dbece294ddc5e7217d5
Vincent Micheli
admin_assigned
2024-07-01T09:46:10.053Z
607d640fc5e1e416b294ea74
/avatars/a660039d2152a2ab8139bc3b4a8cb439.svg
Vincent Micheli
user
vmicheli
66825dbece294ddc5e7217d6
Eloi Alonso
admin_assigned
2024-07-01T09:46:16.330Z
66476d39156169f872095c0c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ja4hmXYNBcprZau6Miux1.png
Eloi Alonso
user
eloialonso
66825dbece294ddc5e7217d7
François Fleuret
admin_assigned
2024-07-01T09:46:22.087Z
6318c37e69ef5467e18fc63f
/avatars/7f1ac593608a89926f4107b4c3dd91d4.svg
François Fleuret
user
francoisfleuret
Efficient World Models with Context-Aware Tokenization
Scaling up deep Reinforcement Learning (RL) methods presents a significant
challenge. Following developments in generative modelling, model-based RL
positions itself as a strong contender. Recent advances in sequence modelling
have led to effective transformer-based world models, albeit at the price of
heavy computations due to the long sequences of tokens required to accurately
simulate environments. In this work, we propose Delta-IRIS, a new agent with
a world model architecture composed of a discrete autoencoder that encodes
stochastic deltas between time steps and an autoregressive transformer that
predicts future deltas by summarizing the current state of the world with
continuous tokens. In the Crafter benchmark, Delta-IRIS sets a new state of
the art at multiple frame budgets, while being an order of magnitude faster to
train than previous attention-based approaches. We release our code and models
at https://github.com/vmicheli/delta-iris.
A new agent, $\Delta$-IRIS, achieves state-of-the-art performance in model-based reinforcement learning by using a discrete autoencoder and autoregressive transformer architecture, resulting in faster training and improved simulation efficiency.
Reinforcement Learning (RL) model-based RL sequence modelling transformer-based world models discrete autoencoder autoregressive transformer Crafter benchmark