title: HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level
    and Fidelity-Rich Conditions in Diffusion Models
summary: We propose an effective method for inserting adapters into text-to-image
  foundation models, which enables the execution of complex downstream tasks
  while preserving the generalization ability of the base model. The core idea of
  this method is to optimize the attention mechanism related to 2D feature maps,
  which enhances the performance of the adapter. This approach was validated on
  the task of meme video generation and achieved significant results. We hope
  this work can provide insights for post-training tasks of large text-to-image
  models. Additionally, as this method demonstrates good compatibility with SD1.5
  derivative models, it holds certain value for the open-source community.
  Therefore, we will release the related code
  (https://songkey.github.io/hellomeme).
paper_authors._id: 6724b544b109aea3270856ea
paper_authors.name: Shengkai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T17:42:50.830Z
paper_authors._id: 63ff47b212c51862e5d9e193
paper_authors.avatarUrl: /avatars/a3d7dee74a73bc56945ac19e8ddcaed9.svg
paper_authors.fullname: Shengkai Zhang
paper_authors.type: user
paper_authors.user: songkey
paper_authors._id: 6724b544b109aea3270856eb
paper_authors.name: Nianhong Jiao
paper_authors._id: 6724b544b109aea3270856ec
paper_authors.name: Tian Li
paper_authors._id: 6724b544b109aea3270856ed
paper_authors.name: Chaojie Yang
paper_authors._id: 6724b544b109aea3270856ee
paper_authors.name: Chenhui Xue
paper_authors._id: 6724b544b109aea3270856ef
paper_authors.name: Boya Niu
paper_authors._id: 6724b544b109aea3270856f0
paper_authors.name: Jun Gao
paper_title: HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level
    and Fidelity-Rich Conditions in Diffusion Models
paper_summary: We propose an effective method for inserting adapters into text-to-image
  foundation models, which enables the execution of complex downstream tasks
  while preserving the generalization ability of the base model. The core idea of
  this method is to optimize the attention mechanism related to 2D feature maps,
  which enhances the performance of the adapter. This approach was validated on
  the task of meme video generation and achieved significant results. We hope
  this work can provide insights for post-training tasks of large text-to-image
  models. Additionally, as this method demonstrates good compatibility with SD1.5
  derivative models, it holds certain value for the open-source community.
  Therefore, we will release the related code
  (https://songkey.github.io/hellomeme).
paper_ai_keywords: adapters text-to-image foundation models downstream tasks attention mechanism 2D feature maps meme video generation