IN THIS ARTICLE
Full source code repository on GitHubGitHub Repo
In this tutorial, we are going to explore one of the most cutting edge technologies in machine learning AI…computer vision! To showcase its capabilities, this step-by-step article will walk you through building your very own desktop security system with a facial recognition machine learning algorithm.
With a simple webcam, your program will be able to recognize the faces of people you choose to allow into the system. It will trigger a text, email, and snapshot image alert system, if any unrecognized faces appear in front of your webcam. We’ll also use Cloudinary and PubNub to build a React Native application that can receive a snapshot image of an “intruder’s” face. It will also allow a user to be added to the system if you desire.
What is Computer Vision?
Computer Vision is a specific field in artificial intelligence that deals with training machine learning models to understand and interpret the visual world. By learning from images and frames from cameras and videos, a computer vision AI can accurately classify the objects it sees and subsequently perform reactionary tasks just like we humans do.
But enough talk…It’s time to code!
Python Code for Your FaceTracking Alert System
Before jumping into the code, be sure you sign up for a free PubNub account so we don’t run into any issues later.
To start building the project from scratch, create your project’s directory using your computer’s command line app:
mkdir faceTrackingApp cd faceTrackingApp
Then create a new Python file called facetracker.py.
Libraries and Dependencies for Computer Vision
OpenCV and face_recognition
First, let’s import some machine learning libraries for our app’s face tracking capabilities. The main libraries we are going to use are OpenCV and face_recognition.
import face_recognition # Machine Learning Library for Face Recognition import cv2 # OpenCV import numpy as np # Handling data import time import os,sys
OpenCV is the most popular machine learning library for realtime Computer Vision. The library has useful tools such as webcam control as well as models to train a face tracking app from scratch.
However, our project will primarily use ageitgey’s face_recognition python library as it already comes with a face recognition model out of the box, making it extremely quick and easy to use.
PubNub
Next, we’re going to setup PubNub as our Data Stream Network to handle all of the data between our Python script and mobile application.
After you’ve retrieved your free PubNub API keys, install the PubNub Python SDK.
pip install 'pubnub>=4.1.4'
Then, import the library in your python file,
from pubnub.callbacks import SubscribeCallback from pubnub.pnconfiguration import PNConfiguration from pubnub.pubnub import PubNub from pubnub.enums import PNOperationType, PNStatusCategory
and configure a PubNub instance with your API keys.
# PubNub Config pnconfig = PNConfiguration() pnconfig.subscribe_key = "YOUR_SUBSCRIBE_KEY" pnconfig.publish_key = "YOUR_PUBLISH_KEY" pnconfig.ssl = False pubnub = PubNub(pnconfig)
Cloudinary
Lastly, we will setup Cloudinary as our Content Delivery Network to store images of intruders’ faces. This will work beautifully with PubNub as our python script can upload the image to Cloudinary, get the URL from the response, then PubNub will send that URL to our Client app to render.
First, sign up for a free Cloudinary account and then install the Cloudinary Python SDK with:
pip install cloudinary
Setup the
CLOUDINARY_URL environment variable by copying it from the Management Console.
Using zsh/bash/sh:
export CLOUDINARY_URL=cloudinary://API-Key:API-Secret@Cloud-name
Import the library in your Python script,
from cloudinary.api import delete_resources_by_tag, resources_by_tag from cloudinary.uploader import upload from cloudinary.utils import cloudinary_url
and configure a Cloudinary instance.
# Cloudinary Config os.chdir(os.path.join(os.path.dirname(sys.argv[0]), '.')) if os.path.exists('settings.py'): exec(open('settings.py').read()) DEFAULT_TAG = "python_sample_basic"
Machine Learning Face Tracking Algorithm
Before we begin building our face recognition machine learning model, we’ll need to declare some global variables:
# Setup some Global Variables video_capture = cv2.VideoCapture(0) # Webcam instance known_face_names = [] # Names of faces known_face_encodings = [] # Encodings of Faces count = 0 # Counter for Number of Unknown Users flag = 0 # Flag for Setting/Unsetting "Intruder Mode"
NOTE: We create a count variable for the unknown users because we are going to dynamically save the user’s snapshot image in a file path. We’re going to append the count to the file name like an ID tag. In order to find this snapshot image later, we need to pull up that user’s count variable, so we can find the image in the file path.
To start training our face recognizer model, we’ll begin with two sample images of faces. You will need two images of two different people’s faces in your project directory.
# Load a sample picture and learn how to recognize it. sample_face_1 = face_recognition.load_image_file("sample_1.jpeg") sample_face_1_encoding = face_recognition.face_encodings(sample_face_1)[0] # Load a second sample picture and learn how to recognize it. sample_face_2 = face_recognition.load_image_file("17.png") sample_face_2_encoding = face_recognition.face_encodings(sample_face_2)[0] # Create arrays of known face encodings and their names known_face_encodings = [ sample_face_1_encoding, sample_face_2_encoding ] # Create Names for Sample Face encodings known_face_names = [ "sample_1", "sample_2" ] # Initialize some variables face_locations = [] face_encodings = [] face_names = [] process_this_frame = True
Next, we will declare a while loop that will run continuously for the duration of the app. The loop will be responsible for the main functions of our app:
- Turning on and displaying the webcam feed
- Tracking faces that appear in front of the webcam and drawing a red box around the face in realtime
- Displaying a name below a known user’s face and “Unknown” for a face that has not been added to the database
- Calling a series of Alerts and Functions to handle when an “Unknown” face appears on screen
while(True): video_capture = cv2.VideoCapture(0) # Grab a single frame of video ret, frame = video_capture.read() # Resize frame of video to 1/4 size for faster face recognition processing small_frame = cv2.resize(frame, (0, 0), fx=0.25, fy=0.25) # Convert the image from BGR color (which OpenCV uses) to RGB color (which face_recognition uses) rgb_small_frame = small_frame[:, :, ::-1] # Only process every other frame of video to save time if process_this_frame: #) #---------------------See next section for this code block's explanation---------------------# ## Set Unknown User Flag and Send Alerts #global flag #if(name=='Unknown' and flag==0): # flag = 1 # Alert() # #--------------------------------------------------------------------------------------------# process_this_frame = not process_this_frame # Display the results for (top, right, bottom, left), name in zip(face_locations, face_names): # Scale back up face locations since the frame we detected in was scaled to 1/4 size top *= 4 right *= 4 bottom *= 4 left *= 4 #(10) & 0xFF == ord('q'): break # Release handle to the webcam video_capture.release() cv2.destroyAllWindows()
Sending Alerts
We shall take care of the case when an unregistered face appears in front of our webcam. We want our program to trigger an alert system the moment it sees an “Unknown” face. Luckily, all we need to do is add a few lines of code to our main while loop at the end of the for face_encoding in face_encodings: loop.
# Set Unknown User Flag and Send Alerts global flag if(name=='Unknown' and flag==0): flag = 1 # Stop repeated calls of Alerts until after the Unknown User is dealt with Alert() # Trigger Alert System
When the alert is triggered, we can then define a function to take a snapshot of the unknown user’s face, call a function to upload the snapshot to Cloudinary, and finally call our Text/Email alert function.
def Alert(): global count video_capture = cv2.VideoCapture(0) # Create Open CV Webcam Instance path = './' # Specify where you want the snapshot to be stored name = 'Unknown_User' + str(count) # Append User ID to File Path # Wait for 3 seconds print('Taking picture in 3') time.sleep(1) print('Taking picture in 2') time.sleep(1) print('Taking picture in 1') time.sleep(1) # Take Picture ret, frame = video_capture.read() # Grayscale Image to save memory space gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) # Save Image in File Path status = cv2.imwrite('% s/% s.jpg' % (path, name),gray) print('Unknown User Saved to Database', status) # Upload Snapshot to Cloudinary upload_files('% s/% s.jpg' % (path,name)) # Send Out Email and Text Alerts sendAlerts()
NOTE: We grayscale the image because the face recognizer doesn’t need color to determine facial features. We also store the snapshot image locally so we can add the face to the recognizer if the client wants to add the user later.
When defining our upload_files() function, we’re passing in the snapshot’s file path so Cloudinary knows where to upload the file from. We then get the response URL of where the image lives in the cloud. We send this url along with a user ID (count of Unknown user) over PubNub to our client application. The client application can then render the image of the snapshot from the Cloudinary URL.
def upload_files(msg): global count # Make global changes to count response = upload(msg, tags=DEFAULT_TAG) # Upload Image to Cloudinary url, options = cloudinary_url( response['public_id'], format=response['format'], width=200, height=150, crop="fill" ) dictionary = {"url": url, "ID": count} pubnub.publish().channel('global').message(dictionary).pn_async(publish_callback) count+=1 # Increment Unknown User Count
In order to publish with PubNub, we need to define a publish callback.
def publish_callback(result, status): pass # Handle PNPublishResult and PNStatus
To setup your Text and Email alerts, you’ll need to sign up for a free ClickSend account as well as a free SendGrid account to get your API keys.
Now you get to see the power and beauty behind PubNub Functions with our Partnered Blocks. Go ahead and visit both our ClickSend Block as well as our SendGrid Block. Through those links, PubNub will automatically generate a customizable PubNub Function.
The serverless, open-source code will completely handle the APIs for you. All you need to do is put in your API keys and you’re good to go!
Once you’ve set up your PubNub Functions, you can define a sendAlerts() function to publish a message, implementing your Text and Email alerts:
def sendAlerts(): dictionary = { "to" : 'RECEIVING PHONE NUMBER', "body": "There is an unregistered user at your desk!" } pubnub.publish().channel('clicksend-text').message(dictionary).pn_async(publish_callback) dictionary = { "to": "EMAIL RECEIVER", "toname": "EMAIL SENDER", "subject": "INTRUDER ALERT", "text": "THERE IS AN UNREGISTERED USER AT YOUR DESK" } pubnub.publish().channel('email-sendgrid-channel').message(dictionary).pn_async(publish_callback)
NOTE: In order to properly use a PubNub block, you need to publish over the same channel specified in your block (you can check this in your blocks Functions dashboard) as well as formatting the message payload properly (according to the block’s documentation).
Adding Users to Our Facetracker
When an Unregistered face is detected on our webcam, our Python script sends an email/text alert as well as a snapshot image to our client application.
We now want to add the ability to add a user’s face to our app’s “known_faces” database, so the user will no longer trigger our alert system. To do this, the client application must publish a message over PubNub.
To receive this message in our python application, we must subscribe to the channel the client is publishing from, and create a Subscriber Callback to handle the incoming message. here # not if status.is_error(): pass # There was an error with the heartbeat operation, handle here else: pass # Heartbeat operation was successful else: pass # Encountered unknown status type def presence(self, pubnub, presence): pass # handle incoming presence data def message(self, pubnub, message): addUser(message.message["ID"], message.message["name"]) pubnub.add_listener(MySubscribeCallback()) pubnub.subscribe().channels('ch1').execute()
NOTE: Above we assume the client is publishing the ID of the unknown user (for image file path) as well as the name of the user (to display below the user’s face).
With the parameters in hand, we can add the new user to our database.
def addUser(ID, name): global known_face_encodings, known_face_names, flag path = './Unknown_User' + str(ID) # Append User ID to File Path # Load User's picture and learn how to recognize it. user_image = face_recognition.load_image_file('% s.jpg' % (path)) # Load Image user_face_encoding = face_recognition.face_encodings(user_image)[0] # Encode Image known_face_encodings.append(user_face_encoding) # Add Encoded Image to 'Known Faces' Array known_face_names.append(name) # Append New User's Name to Database flag = 0 # Reset Unknown User Flag
React Native Code for Our Client Application
Setting Up Our Realtime React Native Environment
Install Xcode so we can build and simulate our app for IOS and Android Studio for Android.
Then install Node.js and watchman using Homebrew:
brew install node brew install watchman
Install the React Native CLI with NPM:
npm install -g react-native-cli
To create a React Native App template, enter the React Native CLI command in your project’s directory:
react-native init client cd client
Since we’re going to be using PubNub in our Client app to send and receive messages, we’ll need to install the PubNub React SDK,
npm install --save pubnub pubnub-react
and then link the library like so:
react-native link pubnub-react
Setting Up Realtime Pub/Sub Messaging
To start sending and receiving messages in realtime in our app, first import the PubNub React SDK.
import PubNubReact from 'pubnub-react';
Then import the TouchableOpacity and Image components from React Native,
import { StyleSheet, View, Text, TextInput, TouchableOpacity, Image, } from 'react-native';
Now we add a constructor at the top of our App Component. The constructor will be responsible for setting up a PubNub instance with our Publish/Subscribe keys as well as initialize the following state variables:
- image – Snapshot image from an unknown user alert (we initialize it with a placeholder image until a Snapshot alert arrives).
- message – Incoming alert message from the face tracking app.
- text – Client user’s input for typing in the name of a user.
- count – To keep track of which unknown user we are getting an alert from.
export default class App extends React.Component { constructor(props) { super(props) this.pubnub = new PubNubReact({ publishKey: "YOUR PUBLISH KEY", subscribeKey: "YOUR SUBSCRIBE KEY" }) //Base State this.state = { image: require('./assets/PLACEHOLDER_IMAGE.jpg'), message: '', text: '', count: 0, } this.pubnub.init(this); } /// .......VVV REST OF THE CODE VVV.......///
When our client app first fires up, we declare an asynchronous function that will subscribe to our face tracking alert channel and handle message events. In this case, we receive the ID (count of unknown user) as well as the snapshot image URL (from Cloudinary) of the unknown user.
async componentDidMount() { this.setUpApp() } async setUpApp(){ this.pubnub.getMessage("global", msg => { this.setState({count: msg.message.ID}) this.setState({image: msg.message.url}) }) this.pubnub.subscribe({ channels: ["global"], withPresence: false }); }
Once that image is received by the mobile app, the client user should then be able to add the unknown user to the face tracker’s “known_faces” database. We can define a function to set the state of the client user’s input for the unknown user’s name.
handleText = (name) => { this.setState({ text: name }) }
We can also write a function to publish the added user’s name along with the added user’s ID.
publishName = (text) => { this.pubnub.publish({ message: { ID: this.state.count, name: text, }, channel: "ch1" }); }
Creating and Rendering App Components
At the top of our screen we’ll render the snapshot image from an incoming “Unknown User” Alert. The source of this image will be a URI we grabbed from the alert message that we saved to state.
<Image source={{uri: this.state.image}} style={{width: 250, height: 250}} />
Below that, we can display a suitable caption.
<Text>{'Do You Know This Person?'}</Text>
We then create a Text Input component to store the name of the User to be added to the face tracker, if the client decides to do so.
<TextInput style = {styles.input} underlineColorAndroid = "transparent" placeholder = "Name" placeholderTextColor = "#9a73ef" autoCapitalize = "none" onChangeText = {this.handleText}/>
Lastly, we create a submit button with TouchableOpacity to publish the added user’s name for our Face Tracker to add to the system:
<TouchableOpacity style = {styles.submitButton} onPress = { () => this.publishName(this.state.text) }> <Text>"SUBMIT"</Text> </TouchableOpacity>
Wrap all those components in a <View> </View> and you’re good to go!
Running the Program
First, start up the React Native client application on Android or iOS by opening a terminal in the client app’s directory.
react-native run-ios or react-native run-android
Then, in another terminal window, run the Python face tracker.
python facetracker.py
If You’re Still Hungry For More…
Feel free to send us any of your questions, concerns, or comments at devrel@pubnub.com.
If you’re still hungry for more PubNub Machine Learning content, here are some other articles that you may be interested in: