title: Customizing Text-to-Image Models with a Single Image Pair
summary: Art reinterpretation is the practice of creating a variation of a reference
  work, making a paired artwork that exhibits a distinct artistic style. We ask
  if such an image pair can be used to customize a generative model to capture
  the demonstrated stylistic difference. We propose Pair Customization, a new
  customization method that learns stylistic difference from a single image pair
  and then applies the acquired style to the generation process. Unlike existing
  methods that learn to mimic a single concept from a collection of images, our
  method captures the stylistic difference between paired images. This allows us
  to apply a stylistic change without overfitting to the specific image content
  in the examples. To address this new task, we employ a joint optimization
  method that explicitly separates the style and content into distinct LoRA
  weight spaces. We optimize these style and content weights to reproduce the
  style and content images while encouraging their orthogonality. During
  inference, we modify the diffusion process via a new style guidance based on
  our learned weights. Both qualitative and quantitative experiments show that
  our method can effectively learn style while avoiding overfitting to image
  content, highlighting the potential of modeling such stylistic differences from
  a single image pair.
paper_authors._id: 66345f93488d269f8f82a068
paper_authors.name: Maxwell Jones
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:49:31.664Z
paper_authors._id: 634e2e38be5a827d4875fd99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666068012901-noauth.png
paper_authors.fullname: Maxwell Jones
paper_authors.type: user
paper_authors.user: maxwelljones14
paper_authors._id: 66345f93488d269f8f82a069
paper_authors.name: Sheng-Yu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:49:38.385Z
paper_authors._id: 62ecaeb1d0d93142d672a067
paper_authors.avatarUrl: /avatars/46a572438c4ac0a0b0d24c7033b8cb53.svg
paper_authors.fullname: Sheng-Yu Wang
paper_authors.type: user
paper_authors.user: sywang
paper_authors._id: 66345f93488d269f8f82a06a
paper_authors.name: Nupur Kumari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:49:50.368Z
paper_authors._id: 62f6a894c3372328414c7021
paper_authors.avatarUrl: /avatars/e8b10912355712f38f10805c31bea962.svg
paper_authors.fullname: Nupur Kumari
paper_authors.type: user
paper_authors.user: nupurkmr9
paper_authors._id: 66345f93488d269f8f82a06b
paper_authors.name: David Bau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:50:18.179Z
paper_authors._id: 6214d6c01e35c843d42d1f77
paper_authors.avatarUrl: /avatars/ac208cd180b4f3ed1ec367e581facfcf.svg
paper_authors.fullname: David Bau
paper_authors.type: user
paper_authors.user: davidbau
paper_authors._id: 66345f93488d269f8f82a06c
paper_authors.name: Jun-Yan Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:50:25.222Z
paper_authors._id: 63a0acc32fabbbb899952a2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
paper_authors.fullname: Jun-Yan Zhu
paper_authors.type: user
paper_authors.user: junyanz
paper_title: Customizing Text-to-Image Models with a Single Image Pair
paper_summary: Art reinterpretation is the practice of creating a variation of a reference
  work, making a paired artwork that exhibits a distinct artistic style. We ask
  if such an image pair can be used to customize a generative model to capture
  the demonstrated stylistic difference. We propose Pair Customization, a new
  customization method that learns stylistic difference from a single image pair
  and then applies the acquired style to the generation process. Unlike existing
  methods that learn to mimic a single concept from a collection of images, our
  method captures the stylistic difference between paired images. This allows us
  to apply a stylistic change without overfitting to the specific image content
  in the examples. To address this new task, we employ a joint optimization
  method that explicitly separates the style and content into distinct LoRA
  weight spaces. We optimize these style and content weights to reproduce the
  style and content images while encouraging their orthogonality. During
  inference, we modify the diffusion process via a new style guidance based on
  our learned weights. Both qualitative and quantitative experiments show that
  our method can effectively learn style while avoiding overfitting to image
  content, highlighting the potential of modeling such stylistic differences from
  a single image pair.
paper_ai_summary: A new method called Pair Customization learns and applies stylistic differences from a single image pair to a generative model without overfitting to specific image content.
paper_ai_keywords: Pair Customization stylistic difference single image pair generative model joint optimization LoRA weight spaces orthogonality diffusion process style guidance