X-Adapter: Adding Universal Compatibility of Plugins for Upgraded
  Diffusion Model
We introduce X-Adapter, a universal upgrader to enable the pretrained
plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
We achieve this goal by training an additional network to control the frozen
upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
frozen copy of the old model to preserve the connectors of different plugins.
Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
from models of different versions for feature remapping. The remapped features
will be used as guidance for the upgraded model. To enhance the guidance
ability of X-Adapter, we employ a null-text training strategy for the upgraded
model. After training, we also introduce a two-stage denoising strategy to
align the initial latents of X-Adapter and the upgraded model. Thanks to our
strategies, X-Adapter demonstrates universal compatibility with various plugins
and also enables plugins of different versions to work together, thereby
expanding the functionalities of diffusion community. To verify the
effectiveness of the proposed method, we conduct extensive experiments and the
results show that X-Adapter may facilitate wider application in the upgraded
foundational diffusion model.
656fe64f2cf29b89e917d1fd
Lingmin Ran
claimed_verified
2023-12-06T16:22:59.015Z
63fb063d1b4b1bd4e7fb3c86
/avatars/241e3b1340bf6450893a821ae73f2f56.svg
Andy
user
Lingmin-Ran
656fe64f2cf29b89e917d1fe
Xiaodong Cun
admin_assigned
2023-12-06T11:37:34.451Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
656fe64f2cf29b89e917d1ff
JiaWei Liu
admin_assigned
2023-12-06T11:37:43.407Z
64230048a73327caad9d8241
/avatars/ac115f1a21c743db7c925f0e18451145.svg
Jiawei Liu
user
JiaweiLIU
656fe64f2cf29b89e917d200
Rui Zhao
admin_assigned
2023-12-06T11:39:10.406Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
656fe64f2cf29b89e917d201
Song Zijie
656fe64f2cf29b89e917d202
Xintao Wang
admin_assigned
2023-12-06T11:39:54.665Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
656fe64f2cf29b89e917d203
Jussi Keppo
656fe64f2cf29b89e917d204
Mike Zheng Shou
admin_assigned
2023-12-06T11:40:40.837Z
63a55320ce5763e06f78519c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
Mike Shou
user
mikeshou
X-Adapter: Adding Universal Compatibility of Plugins for Upgraded
  Diffusion Model
We introduce X-Adapter, a universal upgrader to enable the pretrained
plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
We achieve this goal by training an additional network to control the frozen
upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
frozen copy of the old model to preserve the connectors of different plugins.
Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
from models of different versions for feature remapping. The remapped features
will be used as guidance for the upgraded model. To enhance the guidance
ability of X-Adapter, we employ a null-text training strategy for the upgraded
model. After training, we also introduce a two-stage denoising strategy to
align the initial latents of X-Adapter and the upgraded model. Thanks to our
strategies, X-Adapter demonstrates universal compatibility with various plugins
and also enables plugins of different versions to work together, thereby
expanding the functionalities of diffusion community. To verify the
effectiveness of the proposed method, we conduct extensive experiments and the
results show that X-Adapter may facilitate wider application in the upgraded
foundational diffusion model.
X-Adapter upgrades text-to-image diffusion models like SDXL to work with existing plugins without retraining, through feature remapping and a null-text training strategy.
diffusion model SDXL ControlNet LoRA plug-and-play modules text-to-image mapping layers feature remapping null-text training denoising strategy latent