Real-time Age, Gender and Emotion Prediction from Webcam with Keras and OpenCV
Real-time Age, Gender and Emotion Prediction from Webcam with Keras and OpenCV

Find working codes and trained models here

Chinatown @ Singapore (Photo credit to Lily Banse on Unsplash)

Introduction

In the era of Covid-19, we become more reliant on virtual interactions such as Zoom meetings / Teams chat. These livestream webcam videos have become a rich data source to explore. This article will explore the use case of age, gender and emotion prediction which could facilitate sales person to understand their customers better, for example.

Demo

real-time prediction from my webcam (gif by author)

Overall architecture

Overall implementation structure (image by author)

As shown in above figure, the implementation contains 4 main steps:

Receive input frame from the webcam Identify faces in the webcam and prepare these images for the 3 deep learning models, i.e. age, gender and emotion models Send processed faces to the models and receive prediction outcomes Render prediction outcomes with bounding boxes to screen

In this implementation, we will be using one of the state-of-the-art facial recognition models, MTCNN for step 2. It has a stable Python release based on Keras available at here.

For step 3, we will be training our own customized models. However, for less efforts and better accuracy, you might want to consider transfer learning techniques. Many pre-trained models including, VGG-face, FaceNet, GoogLeNet are available. Note that these pre-trained models might have different input size requirements. Thus faces identified from step 2 need to be processed accordingly.

Facial recognition using MTCNN

Facial recognition has become a mature application of deep learning in recent years. Many algorithms have been proposed for fast and accurate face detection in images / videos. MTCNN is one of these that is based on FaceNet.

In Python’s implementation, the model has been pre-trained and optimized therefore we can make use of the model directly. Nevertheless it’s still important to understand the model’s output.

[

{

'box': [277, 90, 48, 63],

'keypoints':

{

'nose': (303, 131),

'mouth_right': (313, 141),

'right_eye': (314, 114),

'left_eye': (291, 117),

'mouth_left': (296, 143)

},

'confidence': 0.99851983785629272

}

]

for every image, MTCNN returns a list of dictionaries which every dictionary represents a face detected in the photo. Every face is represented as a bounding box — a rectangle that surrounds the face.

box: [x, y, width, height], x and y are coordinates of the top left corner of the bounding box

keypoints: dictionary of facial landmark points detected

confidence: model’s confidence score on the face detected with 1 being the most confident.

Dataset for age/gender/emotion model training

The emotion model is trained from CKPlus Facial Emotion dataset. This dataset contains 981 images from 7 emotion classes: anger, contempt, disgust, fear, happy, sadness and surprise. Each image is in grey scale and has fixed size of 48*48

Image examples from CKPlus dataset (image by author)

The age and gender models are trained from UTKface dataset. The dataset contains more than 20k images. Each image is labelled with age, gender and race. Both full photos and cropped face photos are available for download. For this article, we will be using the full photos and implement our own face alignment method for better accuracy.

Image examples from UTKface dataset (image by author)

Image pre-processing — UTKface dataset

We need to crop face from the full photo using MTCNN or any other facial recognition models. However, most of these algorithms give bounding boxes of varying shapes based on the size and position of the face detected.

Example of MTCNN cropping faces with different sizes (image by author)

Deep learning models require input images to have standardized sizes (caveat: not applicable for fully convolutional network, out-of-scope for this article). Hence resizing of the cropped faces becomes necessary. Direct resizing is the most common and straightforward method with obvious downside — face deformation. As illustrated with the below figure, face becomes significantly wider after direct resizing. This will negatively affect our model performances.

Example of face deformation as the result of direct resizing (image by author)

An ideal cropped face photo should have the face located at the center with no distortion and the required size. If the required size is a square, the following method does the trick.

Obtain facial bounding box from MTCNN Find the center point of the bounding box Find the maximum between the height and width of the bounding box Draw new bounding box based on the center and maximum side length Resize the cropped face from the new bounding box to the required size

If the required size is not a square, step 3 and 4 need to be adjusted to maintain the same ratio as the required sides.

Example of centered resizing (image by author)

For age and gender model, we will use centered resizing methods on the full photos with MTCNN. Required input size is set to be (224, 224, 3) for both models.

Image pre-processing — CKPlus Facial Emotion dataset

It’s not the most ideal dataset for emotion prediction due to its image format (grey scale) and small volume. The advantage is that all images are nicely cropped and aligned thus good for quick prototyping.

One note for this dataset: for each emotion class, individual faces are repeated 3 times. Hence if train/test split is done randomly, target leakage will happen. Recommend to split based on subjects or to remove duplicates before randomly splitting.

Repeated faces in the CKPlus dataset (image by author)

Model structure

Among the three targets, age is the most difficult task. Sometimes even people make mistakes guessing other’s age. Therefore we will need a deeper model for age prediction. In general, these are typical convolutional neural networks. Please note all of these model structures are neither tuned nor optimized. Purpose of the article does not include fine-tuning deep learning models.

Age model

Gender model

Emotion model

Integration with OpenCV explained

Basically, openCV captures video from your webcam (line 2). For every frame, it will convert it to RGB format (line 19). This RGB frame will be sent to detect_face function (line 22), which firstly detects all the faces in the frame using MTCNN and for every face, predicts using the 3 trained models to generate outcomes. These outcomes are returned together with the face bounding box locations (top, right, bottom, left).

OpenCV then makes use of the bounding box locations to draw rectangle on the frame (line 27) and display prediction outcomes in text(line 29 — line 32).

Implementation of the detect_face function can be found in the source code. Note that since emotion model is trained from grey-scale images, RGB image needs to be grey-scaled before being predicted by emotion model.

OpenCV main script

Potential actions for improvements

Better dataset for emotion prediction

Due to computational resource constraint, only 5k images from UTKface dataset were used for age/gender model training. Model performance could be improved by using more images

Image augmentation

Transfer learning / tuning model architecture

Thanks for reading!

Happy New Year!
['Sun Weiran']
['Computer Vision', 'Machine Learning', 'Deep Learning', 'Data Science']