title: X-Adapter: Adding Universal Compatibility of Plugins for Upgraded
    Diffusion Model
summary: We introduce X-Adapter, a universal upgrader to enable the pretrained
  plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
  upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
  We achieve this goal by training an additional network to control the frozen
  upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
  frozen copy of the old model to preserve the connectors of different plugins.
  Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
  from models of different versions for feature remapping. The remapped features
  will be used as guidance for the upgraded model. To enhance the guidance
  ability of X-Adapter, we employ a null-text training strategy for the upgraded
  model. After training, we also introduce a two-stage denoising strategy to
  align the initial latents of X-Adapter and the upgraded model. Thanks to our
  strategies, X-Adapter demonstrates universal compatibility with various plugins
  and also enables plugins of different versions to work together, thereby
  expanding the functionalities of diffusion community. To verify the
  effectiveness of the proposed method, we conduct extensive experiments and the
  results show that X-Adapter may facilitate wider application in the upgraded
  foundational diffusion model.
paper_authors._id: 656fe64f2cf29b89e917d1fd
paper_authors.name: Lingmin Ran
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-06T16:22:59.015Z
paper_authors._id: 63fb063d1b4b1bd4e7fb3c86
paper_authors.avatarUrl: /avatars/241e3b1340bf6450893a821ae73f2f56.svg
paper_authors.fullname: Andy
paper_authors.type: user
paper_authors.user: Lingmin-Ran
paper_authors._id: 656fe64f2cf29b89e917d1fe
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:37:34.451Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 656fe64f2cf29b89e917d1ff
paper_authors.name: JiaWei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:37:43.407Z
paper_authors._id: 64230048a73327caad9d8241
paper_authors.avatarUrl: /avatars/ac115f1a21c743db7c925f0e18451145.svg
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: JiaweiLIU
paper_authors._id: 656fe64f2cf29b89e917d200
paper_authors.name: Rui Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:39:10.406Z
paper_authors._id: 652b83b73b5997ed71a310f2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: ruizhaocv
paper_authors._id: 656fe64f2cf29b89e917d201
paper_authors.name: Song Zijie
paper_authors._id: 656fe64f2cf29b89e917d202
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:39:54.665Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 656fe64f2cf29b89e917d203
paper_authors.name: Jussi Keppo
paper_authors._id: 656fe64f2cf29b89e917d204
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:40:40.837Z
paper_authors._id: 63a55320ce5763e06f78519c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
paper_authors.fullname: Mike Shou
paper_authors.type: user
paper_authors.user: mikeshou
paper_title: X-Adapter: Adding Universal Compatibility of Plugins for Upgraded
    Diffusion Model
paper_summary: We introduce X-Adapter, a universal upgrader to enable the pretrained
  plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
  upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
  We achieve this goal by training an additional network to control the frozen
  upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
  frozen copy of the old model to preserve the connectors of different plugins.
  Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
  from models of different versions for feature remapping. The remapped features
  will be used as guidance for the upgraded model. To enhance the guidance
  ability of X-Adapter, we employ a null-text training strategy for the upgraded
  model. After training, we also introduce a two-stage denoising strategy to
  align the initial latents of X-Adapter and the upgraded model. Thanks to our
  strategies, X-Adapter demonstrates universal compatibility with various plugins
  and also enables plugins of different versions to work together, thereby
  expanding the functionalities of diffusion community. To verify the
  effectiveness of the proposed method, we conduct extensive experiments and the
  results show that X-Adapter may facilitate wider application in the upgraded
  foundational diffusion model.
paper_ai_keywords: X-Adapter pretrained plug-and-play modules ControlNet LoRA text-to-image diffusion model SDXL additional network frozen model trainable mapping layers decoders feature remapping null-text training strategy two-stage denoising strategy initial latents diffusion community foundational diffusion model