title: Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion
    Models
summary: This study explores the role of cross-attention during inference in
  text-conditional diffusion models. We find that cross-attention outputs
  converge to a fixed point after few inference steps. Accordingly, the time
  point of convergence naturally divides the entire inference process into two
  stages: an initial semantics-planning stage, during which, the model relies on
  cross-attention to plan text-oriented visual semantics, and a subsequent
  fidelity-improving stage, during which the model tries to generate images from
  previously planned semantics. Surprisingly, ignoring text conditions in the
  fidelity-improving stage not only reduces computation complexity, but also
  maintains model performance. This yields a simple and training-free method
  called TGATE for efficient generation, which caches the cross-attention output
  once it converges and keeps it fixed during the remaining inference steps. Our
  empirical study on the MS-COCO validation set confirms its effectiveness. The
  source code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.
paper_authors._id: 660e1a88e8763d8d1da50fa6
paper_authors.name: Wentian Zhang
paper_authors._id: 660e1a88e8763d8d1da50fa7
paper_authors.name: Haozhe Liu
paper_authors._id: 660e1a88e8763d8d1da50fa8
paper_authors.name: Jinheng Xie
paper_authors._id: 660e1a88e8763d8d1da50fa9
paper_authors.name: Francesco Faccio
paper_authors._id: 660e1a88e8763d8d1da50faa
paper_authors.name: Mike Zheng Shou
paper_authors._id: 660e1a88e8763d8d1da50fab
paper_authors.name: Jürgen Schmidhuber
paper_title: Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion
    Models
paper_summary: This study explores the role of cross-attention during inference in
  text-conditional diffusion models. We find that cross-attention outputs
  converge to a fixed point after few inference steps. Accordingly, the time
  point of convergence naturally divides the entire inference process into two
  stages: an initial semantics-planning stage, during which, the model relies on
  cross-attention to plan text-oriented visual semantics, and a subsequent
  fidelity-improving stage, during which the model tries to generate images from
  previously planned semantics. Surprisingly, ignoring text conditions in the
  fidelity-improving stage not only reduces computation complexity, but also
  maintains model performance. This yields a simple and training-free method
  called TGATE for efficient generation, which caches the cross-attention output
  once it converges and keeps it fixed during the remaining inference steps. Our
  empirical study on the MS-COCO validation set confirms its effectiveness. The
  source code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.
paper_ai_keywords: cross-attention diffusion models text-conditional inference fixed point semantics-planning stage fidelity-improving stage text-oriented visual semantics image generation computational complexity TGATE training-free method MS-COCO validation set