Learning Flow Fields in Attention for Controllable Person Image
  Generation
Controllable person image generation aims to generate a person image
conditioned on reference images, allowing precise control over the person's
appearance or pose. However, prior methods often distort fine-grained textural
details from the reference image, despite achieving high overall image quality.
We attribute these distortions to inadequate attention to corresponding regions
in the reference image. To address this, we thereby propose learning flow
fields in attention (Leffa), which explicitly guides the target query to attend
to the correct reference key in the attention layer during training.
Specifically, it is realized via a regularization loss on top of the attention
map within a diffusion-based baseline. Our extensive experiments show that
Leffa achieves state-of-the-art performance in controlling appearance (virtual
try-on) and pose (pose transfer), significantly reducing fine-grained detail
distortion while maintaining high image quality. Additionally, we show that our
loss is model-agnostic and can be used to improve the performance of other
diffusion models.
675a9d12154448441ec42806
Zijian Zhou
claimed_verified
2024-12-12T08:26:29.544Z
62e0fcfe4db2175cd2710063
/avatars/0f8d1b5ba3e3eba2af18f98f65f0b180.svg
Zijian Zhou
user
franciszzj
675a9d12154448441ec42807
Shikun Liu
admin_assigned
2024-12-12T09:03:35.757Z
633ef2d6905131c8f52cfb95
/avatars/415f61f80feaffca4e35e029b84ba5d2.svg
Shikun Liu
user
shikunl
675a9d12154448441ec42808
Xiao Han
675a9d12154448441ec42809
Haozhe Liu
675a9d12154448441ec4280a
Kam Woh Ng
675a9d12154448441ec4280b
Tian Xie
675a9d12154448441ec4280c
Yuren Cong
admin_assigned
2024-12-12T09:04:50.005Z
62f8040c47d782a6e286ea8e
/avatars/f45224b851ae79fff9912f6e67159e52.svg
Yuren Cong
user
Yuren
675a9d12154448441ec4280d
Hang Li
675a9d12154448441ec4280e
Mengmeng Xu
admin_assigned
2024-12-12T09:04:56.730Z
650db809ff259b573e42e31d
/avatars/d6e4e4b67cb3ffa197c67197c816139b.svg
Meng Meng Xu
user
Wall-dandelion
675a9d12154448441ec4280f
Juan-Manuel Pérez-Rúa
675a9d12154448441ec42810
Aditya Patel
admin_assigned
2024-12-12T09:05:15.461Z
64534d44e08584d0f5088b2c
/avatars/c5167abd763e186050d2a2e98a1a3b2e.svg
Aditya Patel
user
AdityaPatel
675a9d12154448441ec42811
Tao Xiang
675a9d12154448441ec42812
Miaojing Shi
675a9d12154448441ec42813
Sen He
Learning Flow Fields in Attention for Controllable Person Image
  Generation
Controllable person image generation aims to generate a person image
conditioned on reference images, allowing precise control over the person's
appearance or pose. However, prior methods often distort fine-grained textural
details from the reference image, despite achieving high overall image quality.
We attribute these distortions to inadequate attention to corresponding regions
in the reference image. To address this, we thereby propose learning flow
fields in attention (Leffa), which explicitly guides the target query to attend
to the correct reference key in the attention layer during training.
Specifically, it is realized via a regularization loss on top of the attention
map within a diffusion-based baseline. Our extensive experiments show that
Leffa achieves state-of-the-art performance in controlling appearance (virtual
try-on) and pose (pose transfer), significantly reducing fine-grained detail
distortion while maintaining high image quality. Additionally, we show that our
loss is model-agnostic and can be used to improve the performance of other
diffusion models.
Leffa, a flow field learning method integrated into diffusion models, enhances controllable person image generation by addressing fine-grained detail distortions.
attention layers flow fields diffusion-based models regularization loss virtual try-on pose transfer