readme: ---
  language: en
  license: mit
  library_name: PyTorch
  tags:
  - computer vision
  - GAN
  datasets:
  - multi-pie
  ---
  
  Face Frontalization is a generative computer vision task in which the model takes a photo of a person's head taken at an angle between -90 and 90 degrees, and produces an image of what that person's frontal (i.e. 0 degree) view of the face might look like. The present model was first released in [this repository](https://github.com/scaleway/frontalization) by [Scaleway](https://www.scaleway.com/), a European cloud provider originating from France. It has been previously discussed in a [Scaleway blog post](https://blog.scaleway.com/gpu-instances-using-deep-learning-to-obtain-frontal-rendering-of-facial-images/) and presented at [the DataXDay conference in Paris](https://www.youtube.com/watch?v=aL7rhJz8mAI). The model's GAN architecture was inspired by [the work of R. Huang et al](https://arxiv.org/abs/1704.04086).
  
  # Model description
  
  The Face Frontalization model is the Generator part of a [GAN](https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf) that was trained in a supervised fashion on profile-frontal image pairs. The Discriminator was based on a fairly standard [DCGAN](https://arxiv.org/abs/1511.06434) architecture, where the input is a 128x128x3 image that is processed through multiple convolutional layers, to be classified as either Real or Fake. The Generator had to be modified in order to fit the supervised learning scenario. It consists of convolutional layers (the Encoder of the input image), followed by a 512-dimensional hidden representation that is then fed into the Decoder made up of deconvolutional layers, which produces the output image. For more details on the model's architecture, see [this blog post](https://blog.scaleway.com/gpu-instances-using-deep-learning-to-obtain-frontal-rendering-of-facial-images/).
  
  # Intended uses & limitations
  
  The present Face Frontalization model was not intended to represent the state of the art for this machine learning task. Instead, the goals were:
  
  (a) to demonstrate the benefits of using a GAN for supervised machine learning tasks (whereas the original GAN is an unsupervised generative algorithm; see [this conference talk](https://www.youtube.com/watch?v=aL7rhJz8mAI) for more details);
  
  (b) to show how a complex generative computer vision project can be accomplished on a [Scaleway cloud RENDER-S instance](https://www.scaleway.com/en/gpu-instances/) within ~ a day.
  
  # How to use
  
  The Face Frontalization model is a saved Pytorch model that can be loaded provided the included *network* package is present in the directory. It takes in 3-channel color images resized to 128x128 pixels in the form of [N, 3, 128, 128] tensors (where N is the size of the batch). Ideally, the input images should be closely-cropped photos of faces, taken in good lighting conditions. Here is how the model can be used for inference with a *gradio* image widget, e.g. in a Jupyter notebook:
  
  ```
  import gradio as gr
  import numpy as np
  
  import torch
  from torchvision import transforms
  from torch.autograd import Variable
  
  from PIL import Image
  import matplotlib.pyplot as plt
  
  import warnings
  warnings.filterwarnings('ignore')
  
  # Load the saved Frontalization generator model
  saved_model = torch.load("./generator_v0.pt", map_location=torch.device('cpu'))
  
  def frontalize(image):
     
      # Convert the test image to a [1, 3, 128, 128]-shaped torch tensor
      # (as required by the frontalization model)
      preprocess = transforms.Compose((transforms.ToPILImage(),
                                       transforms.Resize(size = (128, 128)),
                                       transforms.ToTensor()))
      input_tensor = torch.unsqueeze(preprocess(image), 0)
     
      # Use the saved model to generate an output (whose values go between -1 and 1,
      # and this will need to get fixed before the output is displayed)
      generated_image = saved_model(Variable(input_tensor.type('torch.FloatTensor')))
      generated_image = generated_image.detach().squeeze().permute(1, 2, 0).numpy()
      generated_image = (generated_image + 1.0) / 2.0
     
      return generated_image
      
  iface = gr.Interface(frontalize, gr.inputs.Image(type="numpy"), "image")
  iface.launch()    
  ```
  
  # Limitations and bias
  
  As mentioned in the **Intended uses** section, the present model's performance is not intended to compete with the state of the art. Additionally, as the training data had a disproportionately high number of images of caucasian and asian males in their 20s, the model does not perform as well when supplied with images of people not belonging to this limited demographic.
  
  # Training data
  
  The present model was trained on [the CMU Multi-PIE Face Database that is available commercially](https://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html). The input images were closely cropped to include the face of a person photographed at an angle between -90 and 90 degrees. The 