Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning
<|reference_start|>Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning: We introduce Diffusion World Model (DWM), a conditional diffusion model capable of predicting multistep future states and rewards concurrently. As opposed to traditional one-step dynamics models, DWM offers long-horizon predictions in a single forward pass, eliminating the need for recursive queries. We integrate DWM into model-based value estimation, where the short-term return is simulated by future trajectories sampled from DWM. In the context of offline reinforcement learning, DWM can be viewed as a conservative value regularization through generative modeling. Alternatively, it can be seen as a data source that enables offline Q-learning with synthetic data. Our experiments on the D4RL dataset confirm the robustness of DWM to long-horizon simulation. In terms of absolute performance, DWM significantly surpasses one-step dynamics models with a $44\%$ performance gain, and is comparable to or slightly surpassing their model-free counterparts.<|reference_end|>
@article{ding2024diffusion,
    title={Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for
  Offline Reinforcement Learning},
    author={Zihan Ding, Amy Zhang, Yuandong Tian, Qinqing Zheng},
    journal={arXiv preprint arXiv:2402.03570},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03570},
    primaryClass={cs.LG cs.AI}
}