Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated
  Data Refinement Using Contrastive Learning
Although natural language instructions offer an intuitive way to guide
automated image editing, deep-learning models often struggle to achieve
high-quality results, largely due to challenges in creating large, high-quality
training datasets. Previous work has typically relied on text-toimage (T2I)
generative models to produce pairs of original and edited images that simulate
the input/output of an instruction-guided image-editing model. However, these
image pairs often fail to align with the specified edit instructions due to the
limitations of T2I models, which negatively impacts models trained on such
datasets. To address this, we present Instruct-CLIP, a self-supervised method
that learns the semantic changes between original and edited images to refine
and better align the instructions in existing datasets. Furthermore, we adapt
Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it
can be used to train latent diffusion models (LDMs) [19] and efficiently
enforce alignment between the edit instruction and the image changes in latent
space at any step of the diffusion pipeline. We use Instruct-CLIP to correct
the InstructPix2Pix dataset and get over 120K refined samples we then use to
fine-tune their model, guided by our novel Instruct-CLIP-based loss function.
The resulting model can produce edits that are more aligned with the given
instructions. Our code and dataset are available at
https://github.com/SherryXTChen/Instruct-CLIP.git.
67e31b0c3ba899a93859c755
Sherry X. Chen
claimed_verified
2025-03-26T20:45:13.310Z
6350af4d229624a4d4a6bd98
/avatars/2cbc606f23893288ab94b8c8e9b680d3.svg
Sherry Chen
user
SherryXTChen
67e31b0c3ba899a93859c756
Misha Sra
67e31b0c3ba899a93859c757
Pradeep Sen
Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated
  Data Refinement Using Contrastive Learning
Although natural language instructions offer an intuitive way to guide
automated image editing, deep-learning models often struggle to achieve
high-quality results, largely due to challenges in creating large, high-quality
training datasets. Previous work has typically relied on text-toimage (T2I)
generative models to produce pairs of original and edited images that simulate
the input/output of an instruction-guided image-editing model. However, these
image pairs often fail to align with the specified edit instructions due to the
limitations of T2I models, which negatively impacts models trained on such
datasets. To address this, we present Instruct-CLIP, a self-supervised method
that learns the semantic changes between original and edited images to refine
and better align the instructions in existing datasets. Furthermore, we adapt
Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it
can be used to train latent diffusion models (LDMs) [19] and efficiently
enforce alignment between the edit instruction and the image changes in latent
space at any step of the diffusion pipeline. We use Instruct-CLIP to correct
the InstructPix2Pix dataset and get over 120K refined samples we then use to
fine-tune their model, guided by our novel Instruct-CLIP-based loss function.
The resulting model can produce edits that are more aligned with the given
instructions. Our code and dataset are available at
https://github.com/SherryXTChen/Instruct-CLIP.git.
Instruct-CLIP is a self-supervised method for refining instructions in image-editing datasets and aligning them with image changes, specifically for training latent diffusion models.
self-supervised method semantic changes Instruct-CLIP latent diffusion models latent space diffusion pipeline Instruct-CLIP-based loss function