title: Diffusion Policy Policy Optimization
summary: We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic
  framework including best practices for fine-tuning diffusion-based policies
  (e.g. Diffusion Policy) in continuous control and robot learning tasks using
  the policy gradient (PG) method from reinforcement learning (RL). PG methods
  are ubiquitous in training RL policies with other policy parameterizations;
  nevertheless, they had been conjectured to be less efficient for
  diffusion-based policies. Surprisingly, we show that DPPO achieves the
  strongest overall performance and efficiency for fine-tuning in common
  benchmarks compared to other RL methods for diffusion-based policies and also
  compared to PG fine-tuning of other policy parameterizations. Through
  experimental investigation, we find that DPPO takes advantage of unique
  synergies between RL fine-tuning and the diffusion parameterization, leading to
  structured and on-manifold exploration, stable training, and strong policy
  robustness. We further demonstrate the strengths of DPPO in a range of
  realistic settings, including simulated robotic tasks with pixel observations,
  and via zero-shot deployment of simulation-trained policies on robot hardware
  in a long-horizon, multi-stage manipulation task. Website with code:
  diffusion-ppo.github.io
paper_authors._id: 66d80194e1aee6b7147925d9
paper_authors.name: Allen Z. Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T16:21:05.487Z
paper_authors._id: 6392628841491be61b52a4dc
paper_authors.avatarUrl: /avatars/c1e59fbf2252e332a3c86c2bd18afd09.svg
paper_authors.fullname: Allen Z. Ren
paper_authors.type: user
paper_authors.user: allenzren
paper_authors._id: 66d80194e1aee6b7147925da
paper_authors.name: Justin Lidard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T09:21:49.498Z
paper_authors._id: 6570fb5b637aa903549d12d7
paper_authors.avatarUrl: /avatars/992fe518881362bfa71d04987fcccdd0.svg
paper_authors.fullname: Justin Lidard
paper_authors.type: user
paper_authors.user: jlidard
paper_authors._id: 66d80194e1aee6b7147925db
paper_authors.name: Lars L. Ankile
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T15:31:19.092Z
paper_authors._id: 637d214ab61b6d662aeefe09
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d214ab61b6d662aeefe09/DADs1QHxi-Sa7P6Ba25Kw.jpeg
paper_authors.fullname: Lars Lien Ankile
paper_authors.type: user
paper_authors.user: ankile
paper_authors._id: 66d80194e1aee6b7147925dc
paper_authors.name: Anthony Simeonov
paper_authors._id: 66d80194e1aee6b7147925dd
paper_authors.name: Pulkit Agrawal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T09:26:27.275Z
paper_authors._id: 65314176e5b85656bd4a5d72
paper_authors.avatarUrl: /avatars/c62534c0920d90ecb5d4b1506d242e9f.svg
paper_authors.fullname: Pulkit Agrawal
paper_authors.type: user
paper_authors.user: pulkitag
paper_authors._id: 66d80194e1aee6b7147925de
paper_authors.name: Anirudha Majumdar
paper_authors._id: 66d80194e1aee6b7147925df
paper_authors.name: Benjamin Burchfiel
paper_authors._id: 66d80194e1aee6b7147925e0
paper_authors.name: Hongkai Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T09:25:37.305Z
paper_authors._id: 6614c0e00bbea65e71d7efbe
paper_authors.avatarUrl: /avatars/126f3548a15604d28722ff99b6608f8f.svg
paper_authors.fullname: Hongkai Dai
paper_authors.type: user
paper_authors.user: daihongkai
paper_authors._id: 66d80194e1aee6b7147925e1
paper_authors.name: Max Simchowitz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:24:35.152Z
paper_authors._id: 66d89d802c646e71e7eab6bf
paper_authors.avatarUrl: /avatars/008b57235d6056751550dc9a93576e9d.svg
paper_authors.fullname: Max Simchowitz
paper_authors.type: user
paper_authors.user: msimchowitz
paper_title: Diffusion Policy Policy Optimization
paper_summary: We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic
  framework including best practices for fine-tuning diffusion-based policies
  (e.g. Diffusion Policy) in continuous control and robot learning tasks using
  the policy gradient (PG) method from reinforcement learning (RL). PG methods
  are ubiquitous in training RL policies with other policy parameterizations;
  nevertheless, they had been conjectured to be less efficient for
  diffusion-based policies. Surprisingly, we show that DPPO achieves the
  strongest overall performance and efficiency for fine-tuning in common
  benchmarks compared to other RL methods for diffusion-based policies and also
  compared to PG fine-tuning of other policy parameterizations. Through
  experimental investigation, we find that DPPO takes advantage of unique
  synergies between RL fine-tuning and the diffusion parameterization, leading to
  structured and on-manifold exploration, stable training, and strong policy
  robustness. We further demonstrate the strengths of DPPO in a range of
  realistic settings, including simulated robotic tasks with pixel observations,
  and via zero-shot deployment of simulation-trained policies on robot hardware
  in a long-horizon, multi-stage manipulation task. Website with code:
  diffusion-ppo.github.io
paper_ai_summary: DPPO, a policy gradient method for fine-tuning diffusion-based policies, outperforms other RL methods in continuous control and robot learning tasks, offering stable training, robust policies, and effective zero-shot deployment.
paper_ai_keywords: Diffusion Policy Policy Optimization DPPO diffusion-based policies policy gradient reinforcement learning fine-tuning structured exploration on-manifold exploration stable training policy robustness zero-shot deployment simulation-trained policies