CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control
<|reference_start|>CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control: Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules -- a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up. https://guytevet.github.io/CLoSD-page/<|reference_end|>
@article{tevet2024closd:,
    title={CLoSD: Closing the Loop between Simulation and Diffusion for multi-task
  character control},
    author={Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue
  Bin Peng, Amit H. Bermano, Michiel van de Panne},
    journal={arXiv preprint arXiv:2410.03441},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03441},
    primaryClass={cs.CV}
}