readme: ---
  thumbnail: >-
    https://huggingface.co/OFA-Sys/small-stable-diffusion-v0/resolve/main/sample_images_compressed.jpg
  datasets:
  - ChristophSchuhmann/improved_aesthetics_6plus
  license: openrail
  tags:
  - stable-diffusion
  - stable-diffusion-diffusers
  - text-to-image
  language:
  - en
  pipeline_tag: text-to-image
  ---
  
  # Small Stable Diffusion Model Card
  
  Similar image generation quality, but nearly 1/2 smaller!  
  Here are some samples:
  ![Samples](https://huggingface.co/OFA-Sys/small-stable-diffusion-v0/resolve/main/sample_images_compressed.jpg)
  
  
  # Gradio
  
  We support a [Gradio](https://github.com/gradio-app/gradio) Web UI to run small-stable-diffusion-v0:
  [![Open In Spaces](https://camo.githubusercontent.com/00380c35e60d6b04be65d3d94a58332be5cc93779f630bcdfc18ab9a3a7d3388/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f25463025394625413425393725323048756767696e67253230466163652d5370616365732d626c7565)](https://huggingface.co/spaces/akhaliq/small-stable-diffusion-v0)
  
  ## Example
  
  *Use `Diffusers` >=0.8.0, do not support lower versions.*
  
  ```python
  import torch
  from diffusers import StableDiffusionPipeline
  
  
  model_id = "OFA-Sys/small-stable-diffusion-v0/"
  pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
  pipe = pipe.to("cuda")
  
  prompt = "an apple, 4k"
  image = pipe(prompt).images[0]  
      
  image.save("apple.png")
  ```
  
  # Training
  
  ### Initialization 
  
  This model is initialized from stable-diffusion v1-4. As the model structure is not the same as stable-diffusion and the number of parameters is smaller, the parameters of stable diffusion could not be utilized directly. Therefore, small stable diffusion set `layers_per_block=1` and select the first layer of each block in original stable diffusion to initilize the small model.  
  
  ### Training Procedure 
  
  After the initialization, the model have been trained for 1100k steps in 8xA100 GPUS. The training progress consists of three stages. The first stage is a simple pre-training precedure. In the last two stages, the original stable diffusion was utilized to distill knowledge to small model as a teacher model. In all stages, only the parameters in unet were trained and other parameters were frozen.  
  
  
  - **Hardware:** 8 x A100-80GB GPUs
  - **Optimizer:** AdamW
  
  - **Stage 1** - Pretrain the unet part of model.
    - **Steps**: 500,000
    - **Batch:** batch size=8, GPUs=8, Gradient Accumulations=2. Total batch size=128
    - **Learning rate:** warmup to 1e-5 for 10,000 steps and then kept constant
  - **Stage 2** - Distill the model using stable-diffusion v1-4 as the teacher. Besides the ground truth, the training in this stage uses the soft-label (`pred_noise`) generated by teacher model as well.
    - **Steps**: 400,000
    - **Batch:** batch size=8, GPUs=8, Gradient Accumulations=2. Total batch size=128
    - **Learning rate:** warmup to 1e-5 for 5,000 steps and then kept constant
    - **Soft label weight:** 0.5
    - **Hard label weight:** 0.5
  - **Stage 3** - Distill the model using stable-diffusion v1-5 as the teacher. Use several techniques in `Knowledge Distillation of Transformer-based Language Models Revisited`, including similarity-based layer match apart from soft label.
    - **Steps**: 200,000
    - **Batch:** batch size=8, GPUs=8, Gradient Accumulations=2. Total batch size=128
    - **Learning rate:** warmup to 1e-5 for 5,000 steps and then kept constant
    - **Softlabel weight:** 0.5
    - **Hard label weight:** 0.5 
    
  ### Training Data  
  The model developers used the following dataset for training the model:
  1. [LAION-2B en aesthetic](https://huggingface.co/datasets/laion/laion2B-en-aesthetic)
  2. [LAION-Art](https://huggingface.co/datasets/laion/laion-art)
  3. [LAION-HD](https://huggingface.co/datasets/laion/laion-high-resolution)
  
  ### Citation
  
  ```bibtex
  @article{Lu2022KnowledgeDO,
    title={Knowledge Distillation of Transformer-based Language Models Revisited},
    author={Chengqiang Lu and Jianwei Zhang and Yunfei Chu and Zhengyu Chen and Jingren Zhou and Fei Wu and Haiqing Chen and Hongxia Yang},
    journal={ArXiv},
    year={2022},
    volume={abs/2206.14366}
  }
  ```
  
  # Uses
  _The following section is adapted from the [Stable Diffusion model card](https://huggingface.co/CompVis/stable-diffusion-v1-4)_
  
  ## Direct Use
  The model is intended for research purposes only. Possible research areas and
  tasks include
  
  - Safe deployment of models which have the potential to generate harmful content.
  - Probing and understanding the limitations and biases of generative models.
  - Generation of artworks and use in design and other artistic processes.
  - Applications in educational or creative tools.
  - Research on generative models.
  
  Excluded uses are described below.
  
   ### Misuse, Malicious Use, and Out-of-Scope Use
  
  The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content th