ZePo: Zero-Shot Portrait Stylization with Faster Sampling
Diffusion-based text-to-image generation models have significantly advanced
the field of art content synthesis. However, current portrait stylization
methods generally require either model fine-tuning based on examples or the
employment of DDIM Inversion to revert images to noise space, both of which
substantially decelerate the image generation process. To overcome these
limitations, this paper presents an inversion-free portrait stylization
framework based on diffusion models that accomplishes content and style feature
fusion in merely four sampling steps. We observed that Latent Consistency
Models employing consistency distillation can effectively extract
representative Consistency Features from noisy images. To blend the Consistency
Features extracted from both content and style images, we introduce a Style
Enhancement Attention Control technique that meticulously merges content and
style features within the attention space of the target image. Moreover, we
propose a feature merging strategy to amalgamate redundant features in
Consistency Features, thereby reducing the computational load of attention
control. Extensive experiments have validated the effectiveness of our proposed
framework in enhancing stylization efficiency and fidelity. The code is
available at https://github.com/liujin112/ZePo.
66baf3a0a47daa804db162b3
Jin Liu
claimed_verified
2024-08-13T07:37:53.062Z
631afac4838c835180e725eb
/avatars/51e6ec2cf79e04bb078f5ef804ebc394.svg
Jin Liu
user
Jinl
66baf3a0a47daa804db162b4
Huaibo Huang
admin_assigned
2024-08-14T11:08:01.710Z
642259feeaad1bcb28b44f55
/avatars/685791fc7e66d685e9446228c9e278b1.svg
Huaibo Huang
user
Bob072
66baf3a0a47daa804db162b5
Jie Cao
66baf3a0a47daa804db162b6
Ran He
ZePo: Zero-Shot Portrait Stylization with Faster Sampling
Diffusion-based text-to-image generation models have significantly advanced
the field of art content synthesis. However, current portrait stylization
methods generally require either model fine-tuning based on examples or the
employment of DDIM Inversion to revert images to noise space, both of which
substantially decelerate the image generation process. To overcome these
limitations, this paper presents an inversion-free portrait stylization
framework based on diffusion models that accomplishes content and style feature
fusion in merely four sampling steps. We observed that Latent Consistency
Models employing consistency distillation can effectively extract
representative Consistency Features from noisy images. To blend the Consistency
Features extracted from both content and style images, we introduce a Style
Enhancement Attention Control technique that meticulously merges content and
style features within the attention space of the target image. Moreover, we
propose a feature merging strategy to amalgamate redundant features in
Consistency Features, thereby reducing the computational load of attention
control. Extensive experiments have validated the effectiveness of our proposed
framework in enhancing stylization efficiency and fidelity. The code is
available at https://github.com/liujin112/ZePo.
A diffusion-based framework for text-to-image portrait stylization enhances efficiency and fidelity without fine-tuning or DDIM inversion by using Latent Consistency Models and Style Enhancement Attention Control.
diffusion models portrait stylization DDIM Inversion Latent Consistency Models Consistency Features Style Enhancement Attention Control attention space feature merging strategy