title: TweedieMix: Improving Multi-Concept Fusion for Diffusion-based
    Image/Video Generation
summary: Despite significant advancements in customizing text-to-image and video
  generation models, generating images and videos that effectively integrate
  multiple personalized concepts remains a challenging task. To address this, we
  present TweedieMix, a novel method for composing customized diffusion models
  during the inference phase. By analyzing the properties of reverse diffusion
  sampling, our approach divides the sampling process into two stages. During the
  initial steps, we apply a multiple object-aware sampling technique to ensure
  the inclusion of the desired target objects. In the later steps, we blend the
  appearances of the custom concepts in the de-noised image space using Tweedie's
  formula. Our results demonstrate that TweedieMix can generate multiple
  personalized concepts with higher fidelity than existing methods. Moreover, our
  framework can be effortlessly extended to image-to-video diffusion models,
  enabling the generation of videos that feature multiple personalized concepts.
  Results and source code are in our anonymous project page.
paper_authors._id: 670759004341dcee45a674a6
paper_authors.name: Gihyun Kwon
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-10T04:36:37.757Z
paper_authors._id: 66c68b917f4accbaaaaff6cc
paper_authors.avatarUrl: /avatars/40f1a25c393f3cb2e7ae906bb954edac.svg
paper_authors.fullname: Gihyun Kwon
paper_authors.type: user
paper_authors.user: gkwon
paper_authors._id: 670759004341dcee45a674a7
paper_authors.name: Jong Chul Ye
paper_title: TweedieMix: Improving Multi-Concept Fusion for Diffusion-based
    Image/Video Generation
paper_summary: Despite significant advancements in customizing text-to-image and video
  generation models, generating images and videos that effectively integrate
  multiple personalized concepts remains a challenging task. To address this, we
  present TweedieMix, a novel method for composing customized diffusion models
  during the inference phase. By analyzing the properties of reverse diffusion
  sampling, our approach divides the sampling process into two stages. During the
  initial steps, we apply a multiple object-aware sampling technique to ensure
  the inclusion of the desired target objects. In the later steps, we blend the
  appearances of the custom concepts in the de-noised image space using Tweedie's
  formula. Our results demonstrate that TweedieMix can generate multiple
  personalized concepts with higher fidelity than existing methods. Moreover, our
  framework can be effortlessly extended to image-to-video diffusion models,
  enabling the generation of videos that feature multiple personalized concepts.
  Results and source code are in our anonymous project page.
paper_ai_keywords: diffusion models reverse diffusion sampling multiple object-aware sampling de-noised image space Tweedie's formula TweedieMix image-to-video diffusion models