title: Learning Flow Fields in Attention for Controllable Person Image
    Generation
summary: Controllable person image generation aims to generate a person image
  conditioned on reference images, allowing precise control over the person's
  appearance or pose. However, prior methods often distort fine-grained textural
  details from the reference image, despite achieving high overall image quality.
  We attribute these distortions to inadequate attention to corresponding regions
  in the reference image. To address this, we thereby propose learning flow
  fields in attention (Leffa), which explicitly guides the target query to attend
  to the correct reference key in the attention layer during training.
  Specifically, it is realized via a regularization loss on top of the attention
  map within a diffusion-based baseline. Our extensive experiments show that
  Leffa achieves state-of-the-art performance in controlling appearance (virtual
  try-on) and pose (pose transfer), significantly reducing fine-grained detail
  distortion while maintaining high image quality. Additionally, we show that our
  loss is model-agnostic and can be used to improve the performance of other
  diffusion models.
paper_authors._id: 675a9d12154448441ec42806
paper_authors.name: Zijian Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T08:26:29.544Z
paper_authors._id: 62e0fcfe4db2175cd2710063
paper_authors.avatarUrl: /avatars/0f8d1b5ba3e3eba2af18f98f65f0b180.svg
paper_authors.fullname: Zijian Zhou
paper_authors.type: user
paper_authors.user: franciszzj
paper_authors._id: 675a9d12154448441ec42807
paper_authors.name: Shikun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:03:35.757Z
paper_authors._id: 633ef2d6905131c8f52cfb95
paper_authors.avatarUrl: /avatars/415f61f80feaffca4e35e029b84ba5d2.svg
paper_authors.fullname: Shikun Liu
paper_authors.type: user
paper_authors.user: shikunl
paper_authors._id: 675a9d12154448441ec42808
paper_authors.name: Xiao Han
paper_authors._id: 675a9d12154448441ec42809
paper_authors.name: Haozhe Liu
paper_authors._id: 675a9d12154448441ec4280a
paper_authors.name: Kam Woh Ng
paper_authors._id: 675a9d12154448441ec4280b
paper_authors.name: Tian Xie
paper_authors._id: 675a9d12154448441ec4280c
paper_authors.name: Yuren Cong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:04:50.005Z
paper_authors._id: 62f8040c47d782a6e286ea8e
paper_authors.avatarUrl: /avatars/f45224b851ae79fff9912f6e67159e52.svg
paper_authors.fullname: Yuren Cong
paper_authors.type: user
paper_authors.user: Yuren
paper_authors._id: 675a9d12154448441ec4280d
paper_authors.name: Hang Li
paper_authors._id: 675a9d12154448441ec4280e
paper_authors.name: Mengmeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:04:56.730Z
paper_authors._id: 650db809ff259b573e42e31d
paper_authors.avatarUrl: /avatars/d6e4e4b67cb3ffa197c67197c816139b.svg
paper_authors.fullname: Meng Meng Xu
paper_authors.type: user
paper_authors.user: Wall-dandelion
paper_authors._id: 675a9d12154448441ec4280f
paper_authors.name: Juan-Manuel Pérez-Rúa
paper_authors._id: 675a9d12154448441ec42810
paper_authors.name: Aditya Patel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:05:15.461Z
paper_authors._id: 64534d44e08584d0f5088b2c
paper_authors.avatarUrl: /avatars/c5167abd763e186050d2a2e98a1a3b2e.svg
paper_authors.fullname: Aditya Patel
paper_authors.type: user
paper_authors.user: AdityaPatel
paper_authors._id: 675a9d12154448441ec42811
paper_authors.name: Tao Xiang
paper_authors._id: 675a9d12154448441ec42812
paper_authors.name: Miaojing Shi
paper_authors._id: 675a9d12154448441ec42813
paper_authors.name: Sen He
paper_title: Learning Flow Fields in Attention for Controllable Person Image
    Generation
paper_summary: Controllable person image generation aims to generate a person image
  conditioned on reference images, allowing precise control over the person's
  appearance or pose. However, prior methods often distort fine-grained textural
  details from the reference image, despite achieving high overall image quality.
  We attribute these distortions to inadequate attention to corresponding regions
  in the reference image. To address this, we thereby propose learning flow
  fields in attention (Leffa), which explicitly guides the target query to attend
  to the correct reference key in the attention layer during training.
  Specifically, it is realized via a regularization loss on top of the attention
  map within a diffusion-based baseline. Our extensive experiments show that
  Leffa achieves state-of-the-art performance in controlling appearance (virtual
  try-on) and pose (pose transfer), significantly reducing fine-grained detail
  distortion while maintaining high image quality. Additionally, we show that our
  loss is model-agnostic and can be used to improve the performance of other
  diffusion models.
paper_ai_keywords: flow fields attention layer regularization loss attention map diffusion-based baseline virtual try-on pose transfer