title: EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal
    Prompts
summary: Recent advancements in image generation have enabled the creation of
  high-quality images from text conditions. However, when facing multi-modal
  conditions, such as text combined with reference appearances, existing methods
  struggle to balance multiple conditions effectively, typically showing a
  preference for one modality over others. To address this challenge, we
  introduce EMMA, a novel image generation model accepting multi-modal prompts
  built upon the state-of-the-art text-to-image (T2I) diffusion model, ELLA. EMMA
  seamlessly incorporates additional modalities alongside text to guide image
  generation through an innovative Multi-modal Feature Connector design, which
  effectively integrates textual and supplementary modal information using a
  special attention mechanism. By freezing all parameters in the original T2I
  diffusion model and only adjusting some additional layers, we reveal an
  interesting finding that the pre-trained T2I diffusion model can secretly
  accept multi-modal prompts. This interesting property facilitates easy
  adaptation to different existing frameworks, making EMMA a flexible and
  effective tool for producing personalized and context-aware images and even
  videos. Additionally, we introduce a strategy to assemble learned EMMA modules
  to produce images conditioned on multiple modalities simultaneously,
  eliminating the need for additional training with mixed multi-modal prompts.
  Extensive experiments demonstrate the effectiveness of EMMA in maintaining high
  fidelity and detail in generated images, showcasing its potential as a robust
  solution for advanced multi-modal conditional image generation tasks.
paper_authors._id: 666ba936435a489ffd15a8f2
paper_authors.name: Yucheng Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:22:43.785Z
paper_authors._id: 64c38f6a7641afa6ef7d40a9
paper_authors.avatarUrl: /avatars/8277e7c13f80973bdb9a2dae36e6e728.svg
paper_authors.fullname: Han Yucheng
paper_authors.type: user
paper_authors.user: listen2you
paper_authors._id: 666ba936435a489ffd15a8f3
paper_authors.name: Rui Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:22:46.011Z
paper_authors._id: 630443483926de1f7ec6bda0
paper_authors.avatarUrl: /avatars/e8e95ab0d03f58c61846ccc1318502fe.svg
paper_authors.fullname: Rui Wang
paper_authors.type: user
paper_authors.user: budui
paper_authors._id: 666ba936435a489ffd15a8f4
paper_authors.name: Chi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T07:42:51.865Z
paper_authors._id: 64196320ed725fef64419c2a
paper_authors.avatarUrl: /avatars/96feb22fb5e8931d6c9e0ea06148266f.svg
paper_authors.fullname: Chi Zhang
paper_authors.type: user
paper_authors.user: DrChiZhang
paper_authors._id: 666ba936435a489ffd15a8f5
paper_authors.name: Juntao Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:49:22.802Z
paper_authors._id: 664edcade24897b66ca6e5b0
paper_authors.avatarUrl: /avatars/a122c2e56e4927f9e45f125d51d3d836.svg
paper_authors.fullname: Juntao Hu
paper_authors.type: user
paper_authors.user: TBI805
paper_authors._id: 666ba936435a489ffd15a8f6
paper_authors.name: Pei Cheng
paper_authors._id: 666ba936435a489ffd15a8f7
paper_authors.name: Bin Fu
paper_authors._id: 666ba936435a489ffd15a8f8
paper_authors.name: Hanwang Zhang
paper_title: EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal
    Prompts
paper_summary: Recent advancements in image generation have enabled the creation of
  high-quality images from text conditions. However, when facing multi-modal
  conditions, such as text combined with reference appearances, existing methods
  struggle to balance multiple conditions effectively, typically showing a
  preference for one modality over others. To address this challenge, we
  introduce EMMA, a novel image generation model accepting multi-modal prompts
  built upon the state-of-the-art text-to-image (T2I) diffusion model, ELLA. EMMA
  seamlessly incorporates additional modalities alongside text to guide image
  generation through an innovative Multi-modal Feature Connector design, which
  effectively integrates textual and supplementary modal information using a
  special attention mechanism. By freezing all parameters in the original T2I
  diffusion model and only adjusting some additional layers, we reveal an
  interesting finding that the pre-trained T2I diffusion model can secretly
  accept multi-modal prompts. This interesting property facilitates easy
  adaptation to different existing frameworks, making EMMA a flexible and
  effective tool for producing personalized and context-aware images and even
  videos. Additionally, we introduce a strategy to assemble learned EMMA modules
  to produce images conditioned on multiple modalities simultaneously,
  eliminating the need for additional training with mixed multi-modal prompts.
  Extensive experiments demonstrate the effectiveness of EMMA in maintaining high
  fidelity and detail in generated images, showcasing its potential as a robust
  solution for advanced multi-modal conditional image generation tasks.
paper_ai_keywords: EMMA ELLA Multi-modal Feature Connector special attention mechanism text-to-image (T2I) diffusion model pre-trained T2I diffusion model multi-modal prompts personalized context-aware images videos advanced multi-modal conditional image generation tasks