title: StreamMultiDiffusion: Real-Time Interactive Generation with Region-Based
    Semantic Control
summary: The enormous success of diffusion models in text-to-image synthesis has made
  them promising candidates for the next generation of end-user applications for
  image generation and editing. Previous works have focused on improving the
  usability of diffusion models by reducing the inference time or increasing user
  interactivity by allowing new, fine-grained controls such as region-based text
  prompts. However, we empirically find that integrating both branches of works
  is nontrivial, limiting the potential of diffusion models. To solve this
  incompatibility, we present StreamMultiDiffusion, the first real-time
  region-based text-to-image generation framework. By stabilizing fast inference
  techniques and restructuring the model into a newly proposed multi-prompt
  stream batch architecture, we achieve times 10 faster panorama generation
  than existing solutions, and the generation speed of 1.57 FPS in region-based
  text-to-image synthesis on a single RTX 2080 Ti GPU. Our solution opens up a
  new paradigm for interactive image generation named semantic palette, where
  high-quality images are generated in real-time from given multiple hand-drawn
  regions, encoding prescribed semantic meanings (e.g., eagle, girl). Our code
  and demo application are available at
  https://github.com/ironjr/StreamMultiDiffusion.
paper_authors._id: 65f3caf7c4a561cc64d832d0
paper_authors.name: Jaerin Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:25:12.689Z
paper_authors._id: 643bcf4786ab6dbe34ecae59
paper_authors.avatarUrl: /avatars/4fab7726c0eb5f26b2e741d1d6bf2edb.svg
paper_authors.fullname: Jaerin Lee
paper_authors.type: user
paper_authors.user: ironjr
paper_authors._id: 65f3caf7c4a561cc64d832d1
paper_authors.name: Daniel Sungho Jung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T08:03:06.001Z
paper_authors._id: 65601c6ee23401f82005e361
paper_authors.avatarUrl: /avatars/e9fc24bd8c5afd8b07a2f42765d44a7d.svg
paper_authors.fullname: Daniel Sungho Jung
paper_authors.type: user
paper_authors.user: dqj5182
paper_authors._id: 65f3caf7c4a561cc64d832d2
paper_authors.name: Kanggeon Lee
paper_authors._id: 65f3caf7c4a561cc64d832d3
paper_authors.name: Kyoung Mu Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:25:22.867Z
paper_authors._id: 656056b21392aa3beb5de0bd
paper_authors.avatarUrl: /avatars/07f25b750ef308d65f2e6c82506e7816.svg
paper_authors.fullname: Kyoung Mu  Lee 
paper_authors.type: user
paper_authors.user: kyoungmu
paper_title: StreamMultiDiffusion: Real-Time Interactive Generation with Region-Based
    Semantic Control
paper_summary: The enormous success of diffusion models in text-to-image synthesis has made
  them promising candidates for the next generation of end-user applications for
  image generation and editing. Previous works have focused on improving the
  usability of diffusion models by reducing the inference time or increasing user
  interactivity by allowing new, fine-grained controls such as region-based text
  prompts. However, we empirically find that integrating both branches of works
  is nontrivial, limiting the potential of diffusion models. To solve this
  incompatibility, we present StreamMultiDiffusion, the first real-time
  region-based text-to-image generation framework. By stabilizing fast inference
  techniques and restructuring the model into a newly proposed multi-prompt
  stream batch architecture, we achieve times 10 faster panorama generation
  than existing solutions, and the generation speed of 1.57 FPS in region-based
  text-to-image synthesis on a single RTX 2080 Ti GPU. Our solution opens up a
  new paradigm for interactive image generation named semantic palette, where
  high-quality images are generated in real-time from given multiple hand-drawn
  regions, encoding prescribed semantic meanings (e.g., eagle, girl). Our code
  and demo application are available at
  https://github.com/ironjr/StreamMultiDiffusion.
paper_ai_keywords: diffusion models text-to-image synthesis inference time region-based text prompts StreamMultiDiffusion real-time region-based text-to-image generation fast inference techniques diffusion models text-to-image synthesis inference time region-based text prompts StreamMultiDiffusion real-time region-based text-to-image generation fast inference multi-prompt stream batch architecture panorama generation semantic paradigm semantic meanings