README.md

---
library_name: diffusers
license: mit
pipeline_tag: image-to-image
---

# Paint by Inpaint: Learning to Add Image Objects by Removing Them First

The model is designed for instruction-following object addition to images.

We offer four different models:
- **Trained on the PIPE dataset, specifically designed for object addition (This one).**
- The object addition model fine-tuned on a MagicBrush addition subset.
- Trained on the combined PIPE and InstructPix2Pix datasets, intended for general editing.
- The general model fine-tuned on the full MagicBrush dataset.

## Resources

- 💻 [**Visit Project Page**](https://rotsteinnoam.github.io/Paint-by-Inpaint/)

- 📝 [**Read the Paper**](https://arxiv.org/abs/2404.18212)

- 🚀 [**Try Our Demo**](https://huggingface.co/spaces/paint-by-inpaint/demo)

- 🗂️ [**Use PIPE Dataset**](https://huggingface.co/datasets/paint-by-inpaint/PIPE)

#### Running the model

The model is simple to run using the InstructPix2Pix pipeline:

```python
from diffusers import StableDiffusionInstructPix2PixPipeline, EulerAncestralDiscreteScheduler
import torch
import requests
from io import BytesIO
from PIL import Image

model_name = "paint-by-inpaint/add-base"  # addition-base-model
# model_name = "paint-by-inpaint/add-finetuned-mb"  # addition-finetuned-model
# model_name = "paint-by-inpaint/general-base"  # general-base-model
# model_name = "paint-by-inpaint/general-finetuned-mb"  # general-finetuned-model

diffusion_steps = 50
device = "cuda"
image_url = "https://paint-by-inpaint-demo.hf.space/file=/tmp/gradio/99cd3a15aa9bdd3220b4063ebc3ac05e07a611b8/messi.jpeg"
image = Image.open(BytesIO(requests.get(image_url).content)).resize((512, 512))

pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained(model_name, torch_dtype=torch.float16, safety_checker=None).to(device)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)

# Generate the modified image
out_images = pipe(
    "Add a royal silver crown", 
    image=image, 
    guidance_scale=7, 
    image_guidance_scale=1.5, 
    num_inference_steps=diffusion_steps,
    num_images_per_prompt=1
).images

```

## BibTeX

``` Citation
@article{wasserman2024paint,
  title={Paint by Inpaint: Learning to Add Image Objects by Removing Them First},
  author={Wasserman, Navve and Rotstein, Noam and Ganz, Roy and Kimmel, Ron},
  journal={arXiv preprint arXiv:2404.18212},
  year={2024}
}
```