title: DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
summary: Diffusion models have achieved great success in image generation, with the
  backbone evolving from U-Net to Vision Transformers. However, the computational
  cost of Transformers is quadratic to the number of tokens, leading to
  significant challenges when dealing with high-resolution images. In this work,
  we propose Diffusion Mamba (DiM), which combines the efficiency of Mamba, a
  sequence model based on State Space Models (SSM), with the expressive power of
  diffusion models for efficient high-resolution image synthesis. To address the
  challenge that Mamba cannot generalize to 2D signals, we make several
  architecture designs including multi-directional scans, learnable padding
  tokens at the end of each row and column, and lightweight local feature
  enhancement. Our DiM architecture achieves inference-time efficiency for
  high-resolution images. In addition, to further improve training efficiency for
  high-resolution image generation with DiM, we investigate ``weak-to-strong''
  training strategy that pretrains DiM on low-resolution images (256times 256)
  and then finetune it on high-resolution images (512 times 512). We further
  explore training-free upsampling strategies to enable the model to generate
  higher-resolution images (e.g., 1024times 1024 and 1536times 1536)
  without further fine-tuning. Experiments demonstrate the effectiveness and
  efficiency of our DiM.
paper_authors._id: 665010198dba23158e9270da
paper_authors.name: Yao Teng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:31:30.164Z
paper_authors._id: 6427e08288215cee63b1c44d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6427e08288215cee63b1c44d/rzaG978FF-ywzicWNl_xl.jpeg
paper_authors.fullname: yao teng
paper_authors.type: user
paper_authors.user: tytyt
paper_authors._id: 665010198dba23158e9270db
paper_authors.name: Yue Wu
paper_authors._id: 665010198dba23158e9270dc
paper_authors.name: Han Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:19:32.675Z
paper_authors._id: 637751ad7169ed981703d26a
paper_authors.avatarUrl: /avatars/9823f115040a3d6b0240038010e46cb6.svg
paper_authors.fullname: Han Shi
paper_authors.type: user
paper_authors.user: shihan96
paper_authors._id: 665010198dba23158e9270dd
paper_authors.name: Xuefei Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T11:47:13.466Z
paper_authors._id: 641031b1a78453b8d96b8420
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
paper_authors.fullname: Xuefei Ning
paper_authors.type: user
paper_authors.user: Foxfi
paper_authors._id: 665010198dba23158e9270de
paper_authors.name: Guohao Dai
paper_authors._id: 665010198dba23158e9270df
paper_authors.name: Yu Wang
paper_authors._id: 665010198dba23158e9270e0
paper_authors.name: Zhenguo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:31:48.034Z
paper_authors._id: 64f6a67f10a01dc75c6fba70
paper_authors.avatarUrl: /avatars/47989ccce94245bdb1a18722842ce878.svg
paper_authors.fullname: Zhenguo Lee
paper_authors.type: user
paper_authors.user: Zhenguo7
paper_authors._id: 665010198dba23158e9270e1
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:31:27.425Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_title: DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
paper_summary: Diffusion models have achieved great success in image generation, with the
  backbone evolving from U-Net to Vision Transformers. However, the computational
  cost of Transformers is quadratic to the number of tokens, leading to
  significant challenges when dealing with high-resolution images. In this work,
  we propose Diffusion Mamba (DiM), which combines the efficiency of Mamba, a
  sequence model based on State Space Models (SSM), with the expressive power of
  diffusion models for efficient high-resolution image synthesis. To address the
  challenge that Mamba cannot generalize to 2D signals, we make several
  architecture designs including multi-directional scans, learnable padding
  tokens at the end of each row and column, and lightweight local feature
  enhancement. Our DiM architecture achieves inference-time efficiency for
  high-resolution images. In addition, to further improve training efficiency for
  high-resolution image generation with DiM, we investigate ``weak-to-strong''
  training strategy that pretrains DiM on low-resolution images (256times 256)
  and then finetune it on high-resolution images (512 times 512). We further
  explore training-free upsampling strategies to enable the model to generate
  higher-resolution images (e.g., 1024times 1024 and 1536times 1536)
  without further fine-tuning. Experiments demonstrate the effectiveness and
  efficiency of our DiM.
paper_ai_keywords: diffusion models image generation U-Net Vision Transformers State Space Models (SSM) multi-directional scans learnable padding tokens lightweight local feature enhancement inference-time efficiency weak-to-strong training strategy pretraining finetuning training-free upsampling