title: Efficient Diffusion Transformer Policies with Mixture of Expert
    Denoisers for Multitask Learning
summary: Diffusion Policies have become widely used in Imitation Learning, offering
  several appealing properties, such as generating multimodal and discontinuous
  behavior. As models are becoming larger to capture more complex capabilities,
  their computational demands increase, as shown by recent scaling laws.
  Therefore, continuing with the current architectures will present a
  computational roadblock. To address this gap, we propose Mixture-of-Denoising
  Experts (MoDE) as a novel policy for Imitation Learning. MoDE surpasses current
  state-of-the-art Transformer-based Diffusion Policies while enabling
  parameter-efficient scaling through sparse experts and noise-conditioned
  routing, reducing both active parameters by 40% and inference costs by 90% via
  expert caching. Our architecture combines this efficient scaling with
  noise-conditioned self-attention mechanism, enabling more effective denoising
  across different noise levels. MoDE achieves state-of-the-art performance on
  134 tasks in four established imitation learning benchmarks (CALVIN and
  LIBERO). Notably, by pretraining MoDE on diverse robotics data, we achieve 4.01
  on CALVIN ABC and 0.95 on LIBERO-90. It surpasses both CNN-based and
  Transformer Diffusion Policies by an average of 57% across 4 benchmarks, while
  using 90% fewer FLOPs and fewer active parameters compared to default Diffusion
  Transformer architectures. Furthermore, we conduct comprehensive ablations on
  MoDE's components, providing insights for designing efficient and scalable
  Transformer architectures for Diffusion Policies. Code and demonstrations are
  available at https://mbreuss.github.io/MoDE_Diffusion_Policy/.
paper_authors._id: 6763c3475f32892ca1133cc2
paper_authors.name: Moritz Reuss
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:10.717Z
paper_authors._id: 650af93422ce64f22b619549
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/RBybzGRphbiV2MXYHrDoc.png
paper_authors.fullname: Moritz Reuss
paper_authors.type: user
paper_authors.user: mbreuss
paper_authors._id: 6763c3475f32892ca1133cc3
paper_authors.name: Jyothish Pari
paper_authors._id: 6763c3475f32892ca1133cc4
paper_authors.name: Pulkit Agrawal
paper_authors._id: 6763c3475f32892ca1133cc5
paper_authors.name: Rudolf Lioutikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:48:20.326Z
paper_authors._id: 6606fab1d81d611249506cb1
paper_authors.avatarUrl: /avatars/334faf35bd96aee3240a1622c1325459.svg
paper_authors.fullname: Rudolf Lioutikov
paper_authors.type: user
paper_authors.user: lioutikov
paper_title: Efficient Diffusion Transformer Policies with Mixture of Expert
    Denoisers for Multitask Learning
paper_summary: Diffusion Policies have become widely used in Imitation Learning, offering
  several appealing properties, such as generating multimodal and discontinuous
  behavior. As models are becoming larger to capture more complex capabilities,
  their computational demands increase, as shown by recent scaling laws.
  Therefore, continuing with the current architectures will present a
  computational roadblock. To address this gap, we propose Mixture-of-Denoising
  Experts (MoDE) as a novel policy for Imitation Learning. MoDE surpasses current
  state-of-the-art Transformer-based Diffusion Policies while enabling
  parameter-efficient scaling through sparse experts and noise-conditioned
  routing, reducing both active parameters by 40% and inference costs by 90% via
  expert caching. Our architecture combines this efficient scaling with
  noise-conditioned self-attention mechanism, enabling more effective denoising
  across different noise levels. MoDE achieves state-of-the-art performance on
  134 tasks in four established imitation learning benchmarks (CALVIN and
  LIBERO). Notably, by pretraining MoDE on diverse robotics data, we achieve 4.01
  on CALVIN ABC and 0.95 on LIBERO-90. It surpasses both CNN-based and
  Transformer Diffusion Policies by an average of 57% across 4 benchmarks, while
  using 90% fewer FLOPs and fewer active parameters compared to default Diffusion
  Transformer architectures. Furthermore, we conduct comprehensive ablations on
  MoDE's components, providing insights for designing efficient and scalable
  Transformer architectures for Diffusion Policies. Code and demonstrations are
  available at https://mbreuss.github.io/MoDE_Diffusion_Policy/.
paper_ai_keywords: Imitation Learning multimodal behavior constrained behavior computational demands scaling laws parameter-efficient scaling sparse experts noise-conditioned routing active parameters inference costs expert caching noise-conditioned self-attention mechanism denoising super-resolution feature Diffusion Policies MoDE (Mixture-of-Denoising Experts) state-of-the-art performance CALVIN LIBERO FLOPs active parameters comprehensive ablations efficient and scalable Transformer architectures Diffusion Policies