readme: ---
  tags:
  - conditional-image-generation
  - image-to-image
  - gan
  - cyclegan
  # See a list of available tags here:
  # https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts#L12
  # task: unconditional-image-generation or conditional-image-generation or image-to-image
  license: mit
  ---
  
  # CycleGAN for unpaired image-to-image translation. 
  
  ## Model description  
  
  CycleGAN for unpaired image-to-image translation.   
  Given two image domains A and B, the following components are trained end2end to translate between such domains:   
  - A generator A to B, named G_AB conditioned on an image from A   
  - A generator B to A, named G_BA conditioned on an image from B   
  - A domain classifier D_A, associated with G_AB   
  - A domain classifier D_B, associated with G_BA    
  
  
  At inference time, G_AB or G_BA are relevant to translate images, respectively A to B or  B to A.  
  In the general setting, this technique provides style transfer functionalities between the selected image domains A and B.   
  This allows to obtain a generated translation by G_AB, of an image from domain A that resembles the distribution of the images from domain B, and viceversa for the generator G_BA.  
  Under these framework, these aspects have been used to perform style transfer between synthetic data obtained from a simulated driving dataset, GTA5, and the real driving data from Cityscapes.   
  This is of paramount importance to develop autonomous driving perception deep learning models, as this allows to generate synthetic data with automatic annotations which resembles real world images, without requiring the intervention of a human annotator.  
  This is fundamental because a manual annotator has been shown to require 1.5 to 3.3 hours to create semantic and instance segmentation masks for a single images.  
  These have been provided in the original [cityscapes paper (Cordts et al 2016)](https://arxiv.org/abs/2104.13395) and the [adverse condition dataset (Sakaridis et al. 2021)](https://arxiv.org/abs/2104.13395) paper.    
  
    
  Hence the CycleGAN provides forward and backward translation between synthetic and real world data.   
  This has showed to allows high quality translation even in absence of paired sample-ground-truth data.  
  The idea behind such model is that as the synthetic data distribution gets closer to the real world one, deep models do not suffer from degraded performance due to the domain shift issue.   
  A broad literature is available on the minimization of the domain shift, under the research branch of domain adaptation and transfer learning, of which image translation models provide an alternative approach
  
  
  ## Intended uses & limitations
  #### Installation
  ```bash
  git clone https://github.com/huggingface/community-events.git
  cd community-events
  ```
  To install the repository as a python package, run:
  ```bash
  pip install .
  ```  
  
  #### How to use
  
  ```python
  import os
  from PIL import Image
  from torchvision import transforms as T
  from torchvision.transforms import Compose, Resize, ToTensor, Normalize, RandomCrop, RandomHorizontalFlip
  from torchvision.utils import make_grid
  from torch.utils.data import DataLoader
  from huggan.pytorch.cyclegan.modeling_cyclegan import GeneratorResNet
  import torch.nn as nn
  import torch
  import gradio as gr
  import glob
  
  
  
  
  def pred_pipeline(img, transforms):
          orig_shape = img.shape
          input = transforms(img)
          input = input.unsqueeze(0)
          output = model(input)
  
          out_img = make_grid(output,#.detach().cpu(),
                             nrow=1, normalize=True)  
          out_transform = Compose([
              T.Resize(orig_shape[:2]),
              T.ToPILImage()
          ])
          return out_transform(out_img)
  
  
  
  
  n_channels = 3
  image_size = 512
  input_shape = (image_size, image_size)
  
  transform = Compose([
       T.ToPILImage(),
          T.Resize(input_shape),
          ToTensor(),
          Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
      ])
  
  
  model = GeneratorResNet.from_pretrained('Chris1/sim2real', input_shape=(n_channels, image_size, image_size), 
                  num_residual_blocks=9)
                  
  real_images = model(synthetic_images)      
  ```
  
  
  
  #### Limitations and bias
  
  Due to the absence of paired data, some background parts of the synthetic images are seldom wrongly translated, e.g. sky is translated to vegetation.   
  Additional pretext tasks in parallel to the discriminative classifier of fake and real samples could improve the result.  
  One easy improvement is the use of an additional parallel branch that performs semantic segmentation on the synthetic data, in order to learn features which are common to sky and vegetation, thus disentangling their representations as separate classes.    
  
  ## Training data
  
  
  The CycleGAN model is trained on an unpaired dataset of samples from synthetic and real driving data, respectively from the GTA5 and Cityscapes datasets.   
  To this end, the synthetic-to-real dataset can be loaded by means of the function load_dataset