title: IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image
    Diffusion Models
summary: Recent years have witnessed the strong power of large text-to-image diffusion
  models for the impressive generative capability to create high-fidelity images.
  However, it is very tricky to generate desired images using only text prompt as
  it often involves complex prompt engineering. An alternative to text prompt is
  image prompt, as the saying goes: "an image is worth a thousand words".
  Although existing methods of direct fine-tuning from pretrained models are
  effective, they require large computing resources and are not compatible with
  other base models, text prompt, and structural controls. In this paper, we
  present IP-Adapter, an effective and lightweight adapter to achieve image
  prompt capability for the pretrained text-to-image diffusion models. The key
  design of our IP-Adapter is decoupled cross-attention mechanism that separates
  cross-attention layers for text features and image features. Despite the
  simplicity of our method, an IP-Adapter with only 22M parameters can achieve
  comparable or even better performance to a fully fine-tuned image prompt model.
  As we freeze the pretrained diffusion model, the proposed IP-Adapter can be
  generalized not only to other custom models fine-tuned from the same base
  model, but also to controllable generation using existing controllable tools.
  With the benefit of the decoupled cross-attention strategy, the image prompt
  can also work well with the text prompt to achieve multimodal image generation.
  The project page is available at https://ip-adapter.github.io.
paper_authors._id: 64db1fd7978d1f9fafcf862a
paper_authors.name: Hu Ye
paper_authors._id: 64db1fd7978d1f9fafcf862b
paper_authors.name: Jun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-15T13:23:34.942Z
paper_authors._id: 630187a7c1e149ceaffd2401
paper_authors.avatarUrl: /avatars/f13a74ee79f1de08afb3e1475ccf1d86.svg
paper_authors.fullname: Jun Zhang
paper_authors.type: user
paper_authors.user: June20220927
paper_authors._id: 64db1fd7978d1f9fafcf862c
paper_authors.name: Sibo Liu
paper_authors._id: 64db1fd7978d1f9fafcf862d
paper_authors.name: Xiao Han
paper_authors._id: 64db1fd7978d1f9fafcf862e
paper_authors.name: Wei Yang
paper_title: IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image
    Diffusion Models
paper_summary: Recent years have witnessed the strong power of large text-to-image diffusion
  models for the impressive generative capability to create high-fidelity images.
  However, it is very tricky to generate desired images using only text prompt as
  it often involves complex prompt engineering. An alternative to text prompt is
  image prompt, as the saying goes: "an image is worth a thousand words".
  Although existing methods of direct fine-tuning from pretrained models are
  effective, they require large computing resources and are not compatible with
  other base models, text prompt, and structural controls. In this paper, we
  present IP-Adapter, an effective and lightweight adapter to achieve image
  prompt capability for the pretrained text-to-image diffusion models. The key
  design of our IP-Adapter is decoupled cross-attention mechanism that separates
  cross-attention layers for text features and image features. Despite the
  simplicity of our method, an IP-Adapter with only 22M parameters can achieve
  comparable or even better performance to a fully fine-tuned image prompt model.
  As we freeze the pretrained diffusion model, the proposed IP-Adapter can be
  generalized not only to other custom models fine-tuned from the same base
  model, but also to controllable generation using existing controllable tools.
  With the benefit of the decoupled cross-attention strategy, the image prompt
  can also work well with the text prompt to achieve multimodal image generation.
  The project page is available at https://ip-adapter.github.io.
paper_ai_summary: IP-Adapter introduces a lightweight decoupled cross-attention mechanism to enhance pretrained text-to-image diffusion models with image prompt capabilities, enabling multimodal image generation and compatibility with existing tools.
paper_ai_keywords: text-to-image diffusion models image prompt pretrained models decoupled cross-attention mechanism IP-Adapter multimodal image generation