title: Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated
    Data Refinement Using Contrastive Learning
summary: Although natural language instructions offer an intuitive way to guide
  automated image editing, deep-learning models often struggle to achieve
  high-quality results, largely due to challenges in creating large, high-quality
  training datasets. Previous work has typically relied on text-toimage (T2I)
  generative models to produce pairs of original and edited images that simulate
  the input/output of an instruction-guided image-editing model. However, these
  image pairs often fail to align with the specified edit instructions due to the
  limitations of T2I models, which negatively impacts models trained on such
  datasets. To address this, we present Instruct-CLIP, a self-supervised method
  that learns the semantic changes between original and edited images to refine
  and better align the instructions in existing datasets. Furthermore, we adapt
  Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it
  can be used to train latent diffusion models (LDMs) [19] and efficiently
  enforce alignment between the edit instruction and the image changes in latent
  space at any step of the diffusion pipeline. We use Instruct-CLIP to correct
  the InstructPix2Pix dataset and get over 120K refined samples we then use to
  fine-tune their model, guided by our novel Instruct-CLIP-based loss function.
  The resulting model can produce edits that are more aligned with the given
  instructions. Our code and dataset are available at
  https://github.com/SherryXTChen/Instruct-CLIP.git.
paper_authors._id: 67e31b0c3ba899a93859c755
paper_authors.name: Sherry X. Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:45:13.310Z
paper_authors._id: 6350af4d229624a4d4a6bd98
paper_authors.avatarUrl: /avatars/2cbc606f23893288ab94b8c8e9b680d3.svg
paper_authors.fullname: Sherry Chen
paper_authors.type: user
paper_authors.user: SherryXTChen
paper_authors._id: 67e31b0c3ba899a93859c756
paper_authors.name: Misha Sra
paper_authors._id: 67e31b0c3ba899a93859c757
paper_authors.name: Pradeep Sen
paper_title: Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated
    Data Refinement Using Contrastive Learning
paper_summary: Although natural language instructions offer an intuitive way to guide
  automated image editing, deep-learning models often struggle to achieve
  high-quality results, largely due to challenges in creating large, high-quality
  training datasets. Previous work has typically relied on text-toimage (T2I)
  generative models to produce pairs of original and edited images that simulate
  the input/output of an instruction-guided image-editing model. However, these
  image pairs often fail to align with the specified edit instructions due to the
  limitations of T2I models, which negatively impacts models trained on such
  datasets. To address this, we present Instruct-CLIP, a self-supervised method
  that learns the semantic changes between original and edited images to refine
  and better align the instructions in existing datasets. Furthermore, we adapt
  Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it
  can be used to train latent diffusion models (LDMs) [19] and efficiently
  enforce alignment between the edit instruction and the image changes in latent
  space at any step of the diffusion pipeline. We use Instruct-CLIP to correct
  the InstructPix2Pix dataset and get over 120K refined samples we then use to
  fine-tune their model, guided by our novel Instruct-CLIP-based loss function.
  The resulting model can produce edits that are more aligned with the given
  instructions. Our code and dataset are available at
  https://github.com/SherryXTChen/Instruct-CLIP.git.
paper_ai_summary: Instruct-CLIP is a self-supervised method for refining instructions in image-editing datasets and aligning them with image changes, specifically for training latent diffusion models.
paper_ai_keywords: self-supervised method semantic changes Instruct-CLIP latent diffusion models latent space diffusion pipeline Instruct-CLIP-based loss function