title: Efficient Diffusion Policies for Offline Reinforcement Learning
summary: Offline reinforcement learning (RL) aims to learn optimal policies from
  offline datasets, where the parameterization of policies is crucial but often
  overlooked. Recently, Diffsuion-QL significantly boosts the performance of
  offline RL by representing a policy with a diffusion model, whose success
  relies on a parametrized Markov Chain with hundreds of steps for sampling.
  However, Diffusion-QL suffers from two critical limitations. 1) It is
  computationally inefficient to forward and backward through the whole Markov
  chain during training. 2) It is incompatible with maximum likelihood-based RL
  algorithms (e.g., policy gradient methods) as the likelihood of diffusion
  models is intractable. Therefore, we propose efficient diffusion policy (EDP)
  to overcome these two challenges. EDP approximately constructs actions from
  corrupted ones at training to avoid running the sampling chain. We conduct
  extensive experiments on the D4RL benchmark. The results show that EDP can
  reduce the diffusion policy training time from 5 days to 5 hours on
  gym-locomotion tasks. Moreover, we show that EDP is compatible with various
  offline RL algorithms (TD3, CRR, and IQL) and achieves new state-of-the-art on
  D4RL by large margins over previous methods. Our code is available at
  https://github.com/sail-sg/edp.
paper_authors._id: 6477f53ff32a4117fd1786ad
paper_authors.name: Bingyi Kang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:00.429Z
paper_authors._id: 647b5fef6a79fbf5e996c47c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
paper_authors.fullname: Bingyi Kang
paper_authors.type: user
paper_authors.user: bykang
paper_authors._id: 6477f53ff32a4117fd1786ae
paper_authors.name: Xiao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T16:29:46.545Z
paper_authors._id: 6478597d91398856110a6738
paper_authors.avatarUrl: /avatars/c3bc61eb7554ac21946b424e1314f1a7.svg
paper_authors.fullname: Xiao Ma
paper_authors.type: user
paper_authors.user: yusufma555
paper_authors._id: 6477f53ff32a4117fd1786af
paper_authors.name: Chao Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:23:06.815Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 6477f53ff32a4117fd1786b0
paper_authors.name: Tianyu Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T12:31:28.416Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 6477f53ff32a4117fd1786b1
paper_authors.name: Shuicheng Yan
paper_title: Efficient Diffusion Policies for Offline Reinforcement Learning
paper_summary: Offline reinforcement learning (RL) aims to learn optimal policies from
  offline datasets, where the parameterization of policies is crucial but often
  overlooked. Recently, Diffsuion-QL significantly boosts the performance of
  offline RL by representing a policy with a diffusion model, whose success
  relies on a parametrized Markov Chain with hundreds of steps for sampling.
  However, Diffusion-QL suffers from two critical limitations. 1) It is
  computationally inefficient to forward and backward through the whole Markov
  chain during training. 2) It is incompatible with maximum likelihood-based RL
  algorithms (e.g., policy gradient methods) as the likelihood of diffusion
  models is intractable. Therefore, we propose efficient diffusion policy (EDP)
  to overcome these two challenges. EDP approximately constructs actions from
  corrupted ones at training to avoid running the sampling chain. We conduct
  extensive experiments on the D4RL benchmark. The results show that EDP can
  reduce the diffusion policy training time from 5 days to 5 hours on
  gym-locomotion tasks. Moreover, we show that EDP is compatible with various
  offline RL algorithms (TD3, CRR, and IQL) and achieves new state-of-the-art on
  D4RL by large margins over previous methods. Our code is available at
  https://github.com/sail-sg/edp.
paper_ai_summary: Efficient Diffusion Policy (EDP) addresses computational inefficiency and incompatibility issues in offline reinforcement learning by using diffusion models to approximate actions, resulting in faster training and improved performance compared to previous methods.
paper_ai_keywords: diffusion policy offline reinforcement learning Diffsuion-QL parametrized Markov Chain maximum likelihood-based RL policy gradient methods EDP D4RL benchmark gym-locomotion tasks TD3 CRR IQL