How to build an object detection app for sketches?
Weeks ago, I read about the new tool (sketch2code ) from Microsoft that convert sketches to HTML code. So I decided to challenge my self to do the same conversion but for Swift. After less than 20 days the app has been ready for testing and the result was good.

How app work from image selection to conversion to code

To make such a mobile application, we need to follow the next steps :

1- Collect the dataset

2- Image segmentation and data labeling

3- Object detection model

4- OCR

5- Deploy the model

6- Create a conversion algorithm

These processes are basic to build any object detector application you want, you can just replace the dataset. At the end of this tutorial, you will find the full code and important resources.

1- Data Collection

The first thing we need for building any deep learning model is the dataset . So I did some quick search for ready to use dataset, but I didn't find one. So I started drawing some sketches around 10 and downloading other sketches from google images.

Sample of Dataset

Now we need to resize the images to lower resolution to make the training model faster, I find this python code written by Tanner Gilbert.

from PIL import Image

import os import argparse def rescale_images(directory, size):

for img in os.listdir(directory):

im = Image.open(directory+img)

im_resized = im.resize(size, Image.ANTIALIAS)

im_resized.save(directory+img) if __name__ == '__main__':

parser = argparse.ArgumentParser(description="Rescale images")

parser.add_argument('-d', '--directory', type=str,

required=True, help='Directory containing the images')

parser.add_argument('-s', '--size', type=int, nargs=2,

required=True, metavar=('width', 'height'), help='Image size')

args = parser.parse_args()

rescale_images(args.directory, args.size)

Save this script as reseize_images.py in the parent directory of the images then open a terminal at the same folder and type this command :

> python reseize_images.py -d images/ -s 400 600

Before start labeling, we need to randomly split the folder of images to tow. one for training which takes 80% of images another 20% of images for Test folder. You can divide images manually if you have a small dataset, otherwise, you need to use a library like split-folders 0.3.1.

2- Image Segmentation and Labeling

For labeling we will use LabelImg witch is a graphical image annotation tool by drawing bounding boxes around an object in images. We need to download it from GitHub and follow the installation instruction.

After LabelImg interface shows up, you can open the training directory using the “Open Dir” button and draw bounding box by the “Create RectBox” button and write label name then click save. This process needs to be repeated for all images in the training and testing directory.

LabelImg interface

Next, we need to generate a TFRecord which is a file for storing data as a sequence of binary records. This could help us to decrease storage space and training time. In order to create the TFRecords, we need to specify the structure of the data before writing it to the file. So we will download two scripts from Dat Tran’s raccoon detector ( xml_to_csv.py and generate_tfrecord.py). if you open training directory you can see that each image has a corresponding xml file, now will convert these xml files to csv using xml_to_csv.py by making some changes to the main function.

def main():

for folder in ['train', 'test']:

image_path = os.path.join(os.getcwd(), ('images/' +

folder))

xml_df = xml_to_csv(image_path)

xml_df.to_csv(('images/'+folder+'_labels.csv'), index=None)

print('Successfully converted xml to csv.')

In the terminal window type this command at the project folder :

> python xml_to_csv.py

Later, we have to make the transformation, we will open generate_tfrecord.py , update label names in class_text_to_int function :



def class_text_to_int(row_label):

if row_label == ‘CheckBox’:

return 1

elif row_label == ‘TextField’:

return 2

elif row_label == ‘Label’:

return 3

elif row_label == ‘Button’:

return 4

elif row_label == ‘TextView’:

return 5

elif row_label == ‘ImageView’:

return 6

else:

return None

Now the TFRecords can be generated by typing:

> python generate_tfrecord.py — csv_input=images/train_labels.csv — image_dir=images/train — output_path=train.record > python generate_tfrecord.py — csv_input=images/test_labels.csv — image_dir=images/test — output_path=test.record

Finally, we will create a label map where we save the ID for each label. The id should be mach with a number in generate_tfrecord.py .Save it as labelmap.pbtxt

item {

id: 1

name: ‘CheckBox’

}

item {

id: 2

name: ‘TextField’

}

item {

id: 3

name: ‘Label’

}

item {

id: 4

name: ‘Button’

}

item {

id: 5

name: ‘TextView’

}

item {

id: 6

name: ‘ImageView’

}

3- Object Detection Model

We need a model to train our classifier. The Mask Region-based Convolutional Neural Network, or Mask R-CNN, model is one of the state-of-the-art approaches for object recognition tasks. Faster R-CNN, for example, generates proposals (regions) its assumption is that these regions are very likely to contain an object, then does an additional step by classifying which class each region contains. Also, Tensorflow’s Object Detection API is a powerful tool that enables everyone to create their own powerful Image Classifiers.

Type following command in terminal :

Next, we will move train.record and test.record that we have been created before to model/research/object_detection then create a new directory named training and put labelmap.pbtxt in it. Tensorflow’s object detection API comes with many pre-trained models, we will download faster_rcnn_inception_v2_coco you extract the folder in the object detection folder. Then we need the configuration file, as we are using a faster_rcnn_inception model in this project, copy the faster_rcnn_inception_v2_pets.config file from object_detection\samples\configs and paste it in the training directory created before. then I will open it using a text editor in order to change a few lines in the config.

Line 9: change the number of classes to the number of objects you want to detect

num_classes: 6

Line 106: change fine_tune_checkpoint to the path of the model.ckpt file:

fine_tune_checkpoint:"/Users/bdour/Documents/Python/Sketch2Swift/models/research/object_detection/faster_rcnn_inception_v2_coco_2018_01_28/model.ckpt"

Line 123: change input_path to the path of the train.records file:

input_path: "/Users/bdour/Documents/Python/Sketch2Swift/models/research/object_detection/train.record"

Line 135: change input_path to the path of the test.records file:

input_path: "/Users/bdour/Documents/Python/Sketch2Swift/models/research/object_detection/test.record"

Line 125 and 137: change label_map_path to the path of the label map:

label_map_path: "/Users/bdour/Documents/Python/Sketch2Swift/models/research/object_detection/training/labelmap.pbtxt"

Line 130: Set num_examples parameter to the number of images present in the test directory.

num_examples: 7

Let’s train our model ,,

You can find a file named train.py at the location object_detection/legacy/. copy and past it in object_detection directory.

Run the following command to start training.

> python model_main.py --logtostderr --model_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config

Tensorflow creates a checkpoint every 5 minutes and stores it. All the checkpoints are saved in the training directory. The training process could take around 4 hours. If we get lost less than or equal to 0.1, we can terminate the training.

Before moving to the next step, we need to save our trained model. We will prepare an object detection TensorFlow graph for inference using model configuration and a trained checkpoint by keeping all the values of the variables and the Graph in a single file we call this freezing graphs. Freezing a Graph means combining the structure of a model with its weights (checkpoints ). We need to navigate to the training directory and look for the model.ckpt file with the biggest index. Then we can create the inference graph by typing the following command in the command line.

> python export_inference_graph.py

--input_type image_tensor

--pipeline_config_path training/faster_rcnn_inception_v2_pets.config --trained_checkpoint_prefix training/model.ckpt-XXXX

--output_directory inference_graph

XXXX should be filled by the highest checkpoint number.

That’s it, if we go to \object_detection\inference_graph folder we will see our object detection classifier : frozen_inference_graph.pb file.

Time for testing,,

In object_detection folder open new Jupiter notebook, copy and paste some code to download our model and draw boxes in the results of the detection from here . The result should be like this :

3- Optical Character Recognition (OCR)

To detect text inside buttons or labels, we need to apply (OCR) to images using Tesseract. First, we need to install the library :

$ pip install pytesseract

Then, add next line inside visualize_boxes_and_labels_on_image_array function so we can gets a text from image :

text = pytesseract.image_to_string(Image.open(filename))

4- Deploy The Model

There are many good ways to deploy deep learning model in mobile application, I listed here :

But here, we will use a fast easy way that could be used from testing our model in the mobile application by putting trained models into production using Flask API. In object_detection folder , create 2 script to make connection. First one, app.py :



from flask import Flask, jsonify, request

import json

import videoAndImageDetection

app = Flask(__name__)

.route(‘/predict’, methods=[‘POST’])

def predict(): import flaskfrom flask import Flask, jsonify, requestimport jsonimport videoAndImageDetectionapp = Flask(__name__) @app .route(‘/predict’, methods=[‘POST’])def predict(): response = “failure” request_json = request.get_json(force=True) image_data = request_json[‘input’]



responseList = videoAndImageDetection.getjson(image_data) if responseList :

response = json.dumps({‘response’: responseList[0],’response2': responseList[1]})



return response, 200 if __name__ == ‘__main__’:

application.run(debug=True)

The second one, wsgi.py :

from app import app as application

if __name__ == “__main__”:

application.run()

To start connection by type in the command line :

> gunicorn — bind 0.0.0.0:8080 wsgi:application -w 1

5- Create Conversion Algorithm

In Xcode, we create an algorithm that gets a dictionary of all coordinates with their position, size, and text. Then we can easily create any UIKit element programmatically using this information.
['Bdour Ebrahim']
['Deep Learning', 'Object Detection', 'Sketch', 'Swift', 'TensorFlow']