title: ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based
    Image Manipulation
summary: While language-guided image manipulation has made remarkable progress, the
  challenge of how to instruct the manipulation process faithfully reflecting
  human intentions persists. An accurate and comprehensive description of a
  manipulation task using natural language is laborious and sometimes even
  impossible, primarily due to the inherent uncertainty and ambiguity present in
  linguistic expressions. Is it feasible to accomplish image manipulation without
  resorting to external cross-modal language information? If this possibility
  exists, the inherent modality gap would be effortlessly eliminated. In this
  paper, we propose a novel manipulation methodology, dubbed ImageBrush, that
  learns visual instructions for more accurate image editing. Our key idea is to
  employ a pair of transformation images as visual instructions, which not only
  precisely captures human intention but also facilitates accessibility in
  real-world scenarios. Capturing visual instructions is particularly challenging
  because it involves extracting the underlying intentions solely from visual
  demonstrations and then applying this operation to a new image. To address this
  challenge, we formulate visual instruction learning as a diffusion-based
  inpainting problem, where the contextual information is fully exploited through
  an iterative process of generation. A visual prompting encoder is carefully
  devised to enhance the model's capacity in uncovering human intent behind the
  visual instructions. Extensive experiments show that our method generates
  engaging manipulation results conforming to the transformations entailed in
  demonstrations. Moreover, our model exhibits robust generalization capabilities
  on various downstream tasks such as pose transfer, image translation and video
  inpainting.
paper_authors._id: 64cb099ba81988d07338106d
paper_authors.name: Yasheng Sun
paper_authors._id: 64cb099ba81988d07338106e
paper_authors.name: Yifan Yang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-11T10:40:32.277Z
paper_authors._id: 62d18eb81e36881a57f29bf4
paper_authors.avatarUrl: /avatars/104851421b4ee9641daaf15942fa7ea1.svg
paper_authors.fullname: Yif Yang
paper_authors.type: user
paper_authors.user: Yif29
paper_authors._id: 64cb099ba81988d07338106f
paper_authors.name: Houwen Peng
paper_authors._id: 64cb099ba81988d073381070
paper_authors.name: Yifei Shen
paper_authors._id: 64cb099ba81988d073381071
paper_authors.name: Yuqing Yang
paper_authors._id: 64cb099ba81988d073381072
paper_authors.name: Han Hu
paper_authors._id: 64cb099ba81988d073381073
paper_authors.name: Lili Qiu
paper_authors._id: 64cb099ba81988d073381074
paper_authors.name: Hideki Koike
paper_title: ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based
    Image Manipulation
paper_summary: While language-guided image manipulation has made remarkable progress, the
  challenge of how to instruct the manipulation process faithfully reflecting
  human intentions persists. An accurate and comprehensive description of a
  manipulation task using natural language is laborious and sometimes even
  impossible, primarily due to the inherent uncertainty and ambiguity present in
  linguistic expressions. Is it feasible to accomplish image manipulation without
  resorting to external cross-modal language information? If this possibility
  exists, the inherent modality gap would be effortlessly eliminated. In this
  paper, we propose a novel manipulation methodology, dubbed ImageBrush, that
  learns visual instructions for more accurate image editing. Our key idea is to
  employ a pair of transformation images as visual instructions, which not only
  precisely captures human intention but also facilitates accessibility in
  real-world scenarios. Capturing visual instructions is particularly challenging
  because it involves extracting the underlying intentions solely from visual
  demonstrations and then applying this operation to a new image. To address this
  challenge, we formulate visual instruction learning as a diffusion-based
  inpainting problem, where the contextual information is fully exploited through
  an iterative process of generation. A visual prompting encoder is carefully
  devised to enhance the model's capacity in uncovering human intent behind the
  visual instructions. Extensive experiments show that our method generates
  engaging manipulation results conforming to the transformations entailed in
  demonstrations. Moreover, our model exhibits robust generalization capabilities
  on various downstream tasks such as pose transfer, image translation and video
  inpainting.
paper_ai_summary: ImageBrush uses visual instructions learned through diffusion-based inpainting to perform image manipulation without textual guidance, achieving accurate and generalizable results.
paper_ai_keywords: diffusion-based inpainting visual instructions visual prompting encoder pose transfer image translation video inpainting