title: Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models
summary: Transformers have catalyzed advancements in computer vision and natural
  language processing (NLP) fields. However, substantial computational complexity
  poses limitations for their application in long-context tasks, such as
  high-resolution image generation. This paper introduces a series of
  architectures adapted from the RWKV model used in the NLP, with requisite
  modifications tailored for diffusion model applied to image generation tasks,
  referred to as Diffusion-RWKV. Similar to the diffusion with Transformers, our
  model is designed to efficiently handle patchnified inputs in a sequence with
  extra conditions, while also scaling up effectively, accommodating both
  large-scale parameters and extensive datasets. Its distinctive advantage
  manifests in its reduced spatial aggregation complexity, rendering it
  exceptionally adept at processing high-resolution images, thereby eliminating
  the necessity for windowing or group cached operations. Experimental results on
  both condition and unconditional image generation tasks demonstrate that
  Diffison-RWKV achieves performance on par with or surpasses existing CNN or
  Transformer-based diffusion models in FID and IS metrics while significantly
  reducing total computation FLOP usage.
paper_authors._id: 6614b5c67018e413bf5718d7
paper_authors.name: Zhengcong Fei
paper_authors._id: 6614b5c67018e413bf5718d8
paper_authors.name: Mingyuan Fan
paper_authors._id: 6614b5c67018e413bf5718d9
paper_authors.name: Changqian Yu
paper_authors._id: 6614b5c67018e413bf5718da
paper_authors.name: Debang Li
paper_authors._id: 6614b5c67018e413bf5718db
paper_authors.name: Junshi Huang
paper_title: Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models
paper_summary: Transformers have catalyzed advancements in computer vision and natural
  language processing (NLP) fields. However, substantial computational complexity
  poses limitations for their application in long-context tasks, such as
  high-resolution image generation. This paper introduces a series of
  architectures adapted from the RWKV model used in the NLP, with requisite
  modifications tailored for diffusion model applied to image generation tasks,
  referred to as Diffusion-RWKV. Similar to the diffusion with Transformers, our
  model is designed to efficiently handle patchnified inputs in a sequence with
  extra conditions, while also scaling up effectively, accommodating both
  large-scale parameters and extensive datasets. Its distinctive advantage
  manifests in its reduced spatial aggregation complexity, rendering it
  exceptionally adept at processing high-resolution images, thereby eliminating
  the necessity for windowing or group cached operations. Experimental results on
  both condition and unconditional image generation tasks demonstrate that
  Diffison-RWKV achieves performance on par with or surpasses existing CNN or
  Transformer-based diffusion models in FID and IS metrics while significantly
  reducing total computation FLOP usage.
paper_ai_summary: Diffusion-RWKV, an architecture adapted from RWKV for image generation, efficiently handles high-resolution images with reduced computational complexity compared to CNN or Transformer-based models.
paper_ai_keywords: RWKV diffusion model patchnified inputs spatial aggregation complexity FID IS total computation FLOP usage