readme: ---
  license: openrail++
  tags:
  - stable-diffusion
  - text-to-image
  inference: false
  duplicated_from: stabilityai/stable-diffusion-2-inpainting
  ---
  
  # Stable Diffusion v2 Model Card
  This model card focuses on the model associated with the Stable Diffusion v2, available [here](https://github.com/Stability-AI/stablediffusion).
  
  This `stable-diffusion-2-inpainting` model is resumed from [stable-diffusion-2-base](https://huggingface.co/stabilityai/stable-diffusion-2-base) (`512-base-ema.ckpt`) and trained for another 200k steps. Follows the mask-generation strategy presented in [LAMA](https://github.com/saic-mdal/lama) which, in combination with the latent VAE representations of the masked image, are used as an additional conditioning.
  
  ![image](https://huggingface.co/stabilityai/stable-diffusion-2-inpainting/resolve/main/merged-leopards.png)
  
  - Use it with the [`stablediffusion`](https://github.com/Stability-AI/stablediffusion) repository: download the `512-inpainting-ema.ckpt` [here](https://huggingface.co/stabilityai/stable-diffusion-2-inpainting/resolve/main/512-inpainting-ema.ckpt).
  - Use it with 🧨 [`diffusers`](https://huggingface.co/stabilityai/stable-diffusion-2-inpainting#examples)
  
  ## Model Details
  - **Developed by:** Robin Rombach, Patrick Esser
  - **Model type:** Diffusion-based text-to-image generation model
  - **Language(s):** English
  - **License:** [CreativeML Open RAIL++-M License](https://huggingface.co/stabilityai/stable-diffusion-2/blob/main/LICENSE-MODEL)
  - **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses a fixed, pretrained text encoder ([OpenCLIP-ViT/H](https://github.com/mlfoundations/open_clip)).
  - **Resources for more information:** [GitHub Repository](https://github.com/Stability-AI/).
  - **Cite as:**
  
        @InProceedings{Rombach_2022_CVPR,
            author    = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
            title     = {High-Resolution Image Synthesis With Latent Diffusion Models},
            booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
            month     = {June},
            year      = {2022},
            pages     = {10684-10695}
        }
  
  ## Examples
  
  Using the [🤗's Diffusers library](https://github.com/huggingface/diffusers) to run Stable Diffusion 2 inpainting in a simple and efficient manner.
  
  ```bash
  pip install --upgrade git+https://github.com/huggingface/diffusers.git transformers accelerate scipy
  ```
  
  ```python
  from diffusers import StableDiffusionInpaintPipeline
  pipe = StableDiffusionInpaintPipeline.from_pretrained(
      "stable-diffusion-2-inpainting",
      revision="fp16",
      torch_dtype=torch.float16,
  )
  prompt = "Face of a yellow cat, high resolution, sitting on a park bench"
  #image and mask_image should be PIL images.
  #The mask structure is white for inpainting and black for keeping as is
  image = pipe(prompt=prompt, image=image, mask_image=mask_image).images[0]
  image.save("./yellow_cat_on_park_bench.png")
  ```
  
  **Notes**:
  - Despite not being a dependency, we highly recommend you to install [xformers](https://github.com/facebookresearch/xformers) for memory efficient attention (better performance)
  - If you have low GPU RAM available, make sure to add a `pipe.enable_attention_slicing()` after sending it to `cuda` for less VRAM usage (to the cost of speed)
  
  **How it works:**
  `image`          | `mask_image`
  :-------------------------:|:-------------------------:|
  <img src="https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png" alt="drawing" width="300"/> | <img src="https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png" alt="drawing" width="300"/>
  
  
  `prompt`          | `Output`
  :-------------------------:|:-------------------------:|
  <span style="position: relative;bottom: 150px;">Face of a yellow cat, high resolution, sitting on a park bench</span> | <img src="https://huggingface.co/datasets/patrickvonplaten/images/resolve/main/test.png" alt="drawing" width="300"/>
  
  # Uses
  
  ## Direct Use 
  The model is intended for research purposes only. Possible research areas and tasks include
  
  - Safe deployment of models which have the potential to generate harmful content.
  - Probing and understanding the limitations and biases of generative models.
  - Generation of artworks and use in design and other artistic processes.
  - Applications in educational or creative tools.
  - Research on generative models.
  
  Excluded uses are described below.
  
   ### Misuse, Malicious Use, and Out-of-Scope Use
  _Note: This section is originally taken from the [DALLE-MINI model card](https://huggingface.co/dalle-mini/dalle-mini), was used for Stable Diffusion v1, but applies in the same way to Stable Diffusion v2_.
  
  The model s