readme: # VQGAN-CLIP Overview
  
  A repo for running VQGAN+CLIP locally. This started out as a Katherine Crowson VQGAN+CLIP derived Google colab notebook.
  
  <a href="https://replicate.ai/nerdyrodent/vqgan-clip"><img src="https://img.shields.io/static/v1?label=Replicate&message=Demo and Docker Image&color=blue"></a>
  
  Original notebook: [![Open In Colab][colab-badge]][colab-notebook]
  
  [colab-notebook]: <https://colab.research.google.com/drive/1ZAus_gn2RhTZWzOWUpPERNC0Q8OhZRTZ>
  [colab-badge]: <https://colab.research.google.com/assets/colab-badge.svg>
  
  Some example images:
  
  <img src="./samples/Cartoon3.png" width="256px"></img><img src="./samples/Cartoon.png" width="256px"></img><img src="./samples/Cartoon2.png" width="256px"></img>
  <img src="./samples/Bedroom.png" width="256px"></img><img src="./samples/DemonBiscuits.png" width="256px"></img><img src="./samples/Football.png" width="256px"></img>
  <img src="./samples/Fractal_Landscape3.png" width="256px"></img><img src="./samples/Games_5.png" width="256px"></img>
  
  Environment:
  
  * Tested on Ubuntu 20.04
  * GPU: Nvidia RTX 3090
  * Typical VRAM requirements:
    * 24 GB for a 900x900 image
    * 10 GB for a 512x512 image
    * 8 GB for a 380x380 image
  
  You may also be interested in [CLIP Guided Diffusion](https://github.com/nerdyrodent/CLIP-Guided-Diffusion)
  
  ## Set up
  
  This example uses [Anaconda](https://www.anaconda.com/products/individual#Downloads) to manage virtual Python environments.
  
  Create a new virtual Python environment for VQGAN-CLIP:
  
  ```sh
  conda create --name vqgan python=3.9
  conda activate vqgan
  ```
  
  Install Pytorch in the new enviroment:
  
  Note: This installs the CUDA version of Pytorch, if you want to use an AMD graphics card, read the [AMD section below](#using-an-amd-graphics-card).
  
  ```sh
  pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
  ```
  
  Install other required Python packages:
  
  ```sh
  pip install ftfy regex tqdm omegaconf pytorch-lightning IPython kornia imageio imageio-ffmpeg einops torch_optimizer
  ```
  
  Or use the ```requirements.txt``` file, which includes version numbers.
  
  Clone required repositories:
  
  ```sh
  git clone 'https://github.com/nerdyrodent/VQGAN-CLIP'
  cd VQGAN-CLIP
  git clone 'https://github.com/openai/CLIP'
  git clone 'https://github.com/CompVis/taming-transformers'
  ```
  
  Note: In my development environment both CLIP and taming-transformers are present in the local directory, and so aren't present in the `requirements.txt` or `vqgan.yml` files.
  
  As an alternative, you can also pip install taming-transformers and CLIP.
  
  You will also need at least 1 VQGAN pretrained model. E.g.
  
  ```sh
  mkdir checkpoints
  
  curl -L -o checkpoints/vqgan_imagenet_f16_16384.yaml -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fconfigs%2Fmodel.yaml&dl=1' #ImageNet 16384
  curl -L -o checkpoints/vqgan_imagenet_f16_16384.ckpt -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fckpts%2Flast.ckpt&dl=1' #ImageNet 16384
  ```
  Note that users of ```curl``` on Microsoft Windows should use double quotes.
  
  The `download_models.sh` script is an optional way to download a number of models. By default, it will download just 1 model.
  
  See <https://github.com/CompVis/taming-transformers#overview-of-pretrained-models> for more information about VQGAN pre-trained models, including download links.
  
  By default, the model .yaml and .ckpt files are expected in the `checkpoints` directory.
  See <https://github.com/CompVis/taming-transformers> for more information on datasets and models.
  
  Video guides are also available:
  * Linux - https://www.youtube.com/watch?v=1Esb-ZjO7tw
  * Windows - https://www.youtube.com/watch?v=XH7ZP0__FXs
  
  ### Using an AMD graphics card
  
  Note: This hasn't been tested yet.
  
  ROCm can be used for AMD graphics cards instead of CUDA. You can check if your card is supported here:
  <https://github.com/RadeonOpenCompute/ROCm#supported-gpus>
  
  Install ROCm accordng to the instructions and don't forget to add the user to the video group:
  <https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html>
  
  The usage and set up instructions above are the same, except for the line where you install Pytorch.
  Instead of `pip install torch==1.9.0+cu111 ...`, use the one or two lines which are displayed here (select Pip -> Python-> ROCm):
  <https://pytorch.org/get-started/locally/>
  
  ### Using the CPU
  
  If no graphics card can be found, the CPU is automatically used and a warning displayed.
  
  Regardless of an available graphics card, the CPU can also be used by adding this command line argument: `-cd cpu`
  
  This works with the CUDA version of Pytorch, even without CUDA drivers installed, but doesn't seem to work with ROCm as of now.
  
  ### Uninstalling
  
  Remove the Python enviroment:
  
  ```sh
  conda remove --name vqgan --all
  ```
  
  and delete the `VQGAN-CLIP` directory.
  
  ## Run
  
  To generate images from text, specify your text prompt as shown in the example below:
  
  ```sh
  pytho