<file_sep>**Artificial Intelligence Frameworks:**
-  Frameworks:
  - Caffee
  - Theano
  - PyTroch
  - Distbelief
- Tensorflow: Set of APIs and data structures


**Website for datasets:**
- seach for datasets:
  - https://www.kaggle.com/datasets
  - https://cocodataset.org/#explore
  - http://www.image-net.org/
  - https://archive.ics.uci.edu/ml/datasets.php
  - http://yann.lecun.com/exdb/mnist/
- search for websites: 
  - https://datasetsearch.research.google.com/
  - https://www.kdnuggets.com/datasets/index.html

**Pre-Trained Networks in Keras:**
- VGG16
- VGG19
- Resnet50
- Inception V3
- Xception
- MobileNet

**Object Detection Approaches:**
- RCNN (https://arxiv.org/abs/1311.2524)
- Fast RCNN (https://arxiv.org/abs/1504.08083)
- Faster RCNN (https://arxiv.org/abs/1506.01497)
- Yolo - you only look once
- Implementation unsing TensorFlow
- SSD (https://arxiv.org/pdf/1512.02325.pdf):
  - Key challange: number of objects is unknown (How many bounding boxes? How many classifictions?)
    - proviode a fixed number of bounding boxes+classifications(maximum)
    - classify bounding boxes as "object" or "not an object" -> only considering "objects" to produce variable number of boxes+classifications
  - Training:
    - True classifications and bounding boxes are known
    - Secify the output:
      - Split imput data into 3x3 cells
        - for each cell:
         
            <img src="https://github.com/gitkatrin/gesture_project/blob/master/images/Training_vector.PNG" width="250">
         
        - Cell that contains the center point of the object is associated to the object:
          -> identify cell with center point of bounding box
          -> conpare bounding box with different anchor boxes in this cell
          -> associate object to ancher box with most sililar shape (higherst IoU)
        - Multiple anchor boxes in each cell (anchor box: initial guess for a bounding box with fixed size)
          -> output one y for every anchor box
  - Testing:
    - basic probelm: obtain too many bounding boxes; trainined network would output a vector y for every anchor box
    - remove all bounding boxes which are p < 0,5 (p: Probability that its an object)
      - normally left with a few bounding boxes for each object (many anchor boxes overlap with object because of the 19x19 grid)
    - Non-maximum suppression:
      1. Remove all anchor boxes: p < 0,5
      2. Find anchor box with the largest probability of being an object and store that to set of object detections (bounding box i)
      3. Remove all anchor boxes:
          - the same most probable class is the same than bpunding box i
          - bounding box overlaps substantially with bounding box i 
      4. Repeat until fixed bounding boxes
  - **Paper:** 
    - The convolutional model for predicting detections is different for each feature layer
    - Model: Default boxes and aspect ratios:
      - associate set of default bounding boxes with each feature map cell (one cell with set of bounding boxes) for multiple feature maps at the top of the network
      - default boxes tile the feature map in a convolutional manner -> position of each box is fixed to corresponding cell
      - in each feature map cell: 
        - predict **offsets relative** to the default box shapes
        - per-class **scores** that indicate the presence of a class instance (in each box)
      - compute relative to original default box shape, for each box out of k at a given location the c class scores and 4 offsets
      
          -> (c + 4)k filters, applied around each location in the feature map -> (c + 4)kmn outputs (for m x n feature map)
      - like anchor boxes (Faster R-CNN) but apply them on several feature maps of different resolutions
      - allowing different default box shapes in severat feature maps -> discretize the space of possible output box shapes
    - Training:
      - ground truth information needs to be assigned to specific outputs in the fixed set of detector outputs
      - assignment determined -> loss function and back propagation are applied end-to-end
      - choosing the set of default boxes and scales for detection
      - scales for detection hard negative mining and data augmentation strategies
      - matching strategy:
        - 
<file_sep>#!/usr/bin/env python
# coding: utf-8
"""
Object Detection (On Image) From TF2 Saved Model
=====================================
"""

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'    # Suppress TensorFlow logging (1)
import pathlib
import tensorflow as tf
import cv2
import argparse

tf.get_logger().setLevel('ERROR')           # Suppress TensorFlow logging (2)

parser = argparse.ArgumentParser()
parser.add_argument('--model', help='Folder that the Saved Model is Located In',
                    default='exported-models/my_mobilenet_model')
parser.add_argument('--labels', help='Where the Labelmap is Located',
                    default='exported-models/my_mobilenet_model/saved_model/label_map.pbtxt')
parser.add_argument('--image', help='Name of the single image to perform detection on',
                    default='JENGA_LIVINGROOM_S_T_frame_1650.jpg')
parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects',
                    default=0.3)
                    
args = parser.parse_args()
# Enable GPU dynamic memory allocation
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

# PROVIDE PATH TO IMAGE DIRECTORY
IMAGE_PATHS = args.image


# PROVIDE PATH TO MODEL DIRECTORY
PATH_TO_MODEL_DIR = args.model

# PROVIDE PATH TO LABEL MAP
PATH_TO_LABELS = args.labels

# PROVIDE THE MINIMUM CONFIDENCE THRESHOLD
MIN_CONF_THRESH = float(args.threshold)

# LOAD THE MODEL

import time
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as viz_utils

PATH_TO_SAVED_MODEL = PATH_TO_MODEL_DIR + "/saved_model"

print('Loading model...', end='')
start_time = time.time()

# LOAD SAVED MODEL AND BUILD DETECTION FUNCTION
detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)

end_time = time.time()
elapsed_time = end_time - start_time
print('Done! Took {} seconds'.format(elapsed_time))

# LOAD LABEL MAP DATA FOR PLOTTING

category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS,
                                                                    use_display_name=True)
import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')   # Suppress Matplotlib warnings
from collections import namedtuple

def load_image_into_numpy_array(path):
    """Load an image from file into a numpy array.

    Puts image into numpy array to feed into tensorflow graph.
    Note that by convention we put it into a numpy array with shape
    (height, width, channels), where channels=3 for RGB.

    Args:
      path: the file path to the image

    Returns:
      uint8 numpy array with shape (img_height, img_width, 3)
    """
    return np.array(Image.open(path))




print('Running inference for {}... '.format(IMAGE_PATHS), end='')
#==========================================================================
# define the `Detection` object (gt= ground truth, pred= prediction)
Detection = namedtuple("Detection", ["image_path", "gt", "pred"])

def batch_iou(boxA, boxB):
    
    # COORDINATES OF THE INTERSECTION BOXES
    xA = max(boxA[0], boxB[0])
    yA = max(boxA[1], boxB[1])
    xB = min(boxA[2], boxB[2])
    yB = min(boxA[3], boxB[3])

    # compute the area of intersection rectangle
    interArea = max(0, xB - xA + 1) * max(0, yB - yA + 1)
    
    # compute the area of both the prediction and ground-truth
    # rectangles
    boxAArea = (boxA[2] - boxA[0] + 1) * (boxA[3] - boxA[1] + 1)
    boxBArea = (boxB[2] - boxB[0] + 1) * (boxB[3] - boxB[1] + 1)
    
    # compute the intersection over union by taking the intersection
    # area and dividing it by the sum of prediction + ground-truth
    # areas - the interesection area
    iou = interArea / float(boxAArea + boxBArea - interArea)
    
    # return the intersection over union value
    return iou


# hier die csv einlesen !!!!!!!!!!!!!!!!!
    
# define the list of example detections
examples = [
        Detection("JENGA_LIVINGROOM_S_T_frame_1650.jpg", [633, 383, 845, 578], [630, 380, 842, 575])]

# loop over the example detections
#for detection in examples:
#    # load the image
#    image = cv2.imread(detection.image_path)
#	# draw the ground-truth bounding box along with the predicted
#	# bounding box
#    cv2.rectangle(image, tuple(detection.gt[:2]), 
#                  tuple(detection.gt[2:]), (0, 255, 0), 2)
#    cv2.rectangle(image, tuple(detection.pred[:2]), 
#                  tuple(detection.pred[2:]), (0, 0, 255), 2)
#    # compute the intersection over union and display it
#    iou = batch_iou(detection.gt, detection.pred)
#    cv2.putText(image, "IoU: {:.4f}".format(iou), (10, 30),
#                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
#    print("{}: {:.4f}".format(detection.image_path, iou))
#	# show the output image
#    cv2.imshow("Image", image)
#    cv2.waitKey(0)
#============================================================================

image = cv2.imread(IMAGE_PATHS)
image_rgb = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
imH, imW, _ = image.shape
image_expanded = np.expand_dims(image_rgb, axis=0)

# The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
input_tensor = tf.convert_to_tensor(image)
# The model expects a batch of images, so add an axis with `tf.newaxis`.
input_tensor = input_tensor[tf.newaxis, ...]

# input_tensor = np.expand_dims(image_np, 0)
detections = detect_fn(input_tensor)

# All outputs are batches tensors.
# Convert to numpy arrays, and take index [0] to remove the batch dimension.
# We're only interested in the first num_detections.
num_detections = int(detections.pop('num_detections'))
detections = {key: value[0, :num_detections].numpy()
               for key, value in detections.items()}
detections['num_detections'] = num_detections

# detection_classes should be ints.
detections['detection_classes'] = detections['detection_classes'].astype(np.int64)
scores = detections['detection_scores']
boxes = detections['detection_boxes']
classes = detections['detection_classes']
count = 0
for i in range(len(scores)):
    if ((scores[i] > MIN_CONF_THRESH) and (scores[i] <= 1.0)):
        #increase count
        count += 1
        # Get bounding box coordinates and draw box
        # Interpreter can return coordinates that are outside of image dimensions, need to force them to be within image using max() and min()
        ymin = int(max(1,(boxes[i][0] * imH)))
        xmin = int(max(1,(boxes[i][1] * imW)))
        ymax = int(min(imH,(boxes[i][2] * imH)))
        xmax = int(min(imW,(boxes[i][3] * imW)))
        
        cv2.rectangle(image, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2)
        # Draw label
        object_name = category_index[int(classes[i])]['name'] # Look up object name from "labels" array using class index
        label = '%s: %d%%' % (object_name, int(scores[i]*100)) # Example: 'person: 72%'
        labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) # Get font size
        label_ymin = max(ymin, labelSize[1] + 10) # Make sure not to draw label too close to top of window
        cv2.rectangle(image, (xmin, label_ymin-labelSize[1]-10), (xmin+labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED) # Draw white box to put label text in
        cv2.putText(image, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2) # Draw label text
        
        pred_box = np.array([xmin, ymin, xmax, ymax])
        
        
        truth_box = np.array([633, 383, 845, 578])
        cv2.rectangle(image, (633,383), (845,578), (255, 0, 0), 2)
        print(pred_box, truth_box)
        
        
        iou = batch_iou(truth_box, pred_box)
        #cv2.rectangle(image, (xmax, label_ymin-labelSize[1]-10), (xmax+labelSize[4], label_ymin+baseLine-10), (0, 0, 0), cv2.FILLED) # Draw white box to put label text in
        cv2.putText(image, "IoU: {:.4f}".format(iou), (xmax, label_ymin-7),
                cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 0, 0), 2)
    

cv2.putText (image,'Total Detections : ' + str(count),(10,25),cv2.FONT_HERSHEY_SIMPLEX,1,(70,235,52),2,cv2.LINE_AA)
print('Done')
# DISPLAYS OUTPUT IMAGE
cv2.imshow('Object Counter', image)
# CLOSES WINDOW ONCE KEY IS PRESSED
cv2.waitKey(0)
# CLEANUP
cv2.destroyAllWindows()

<file_sep>#!/usr/bin/env python
# coding: utf-8
"""
Object Detection (On Image) From TF2 Saved Model
=====================================
"""

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'    # Suppress TensorFlow logging (1)
import pathlib
import tensorflow as tf
import cv2
import argparse

tf.get_logger().setLevel('ERROR')           # Suppress TensorFlow logging (2)

parser = argparse.ArgumentParser()
parser.add_argument('--model', help='Folder that the Saved Model is Located In',
                    default='exported-models/my_mobilenet_model')
parser.add_argument('--labels', help='Where the Labelmap is Located',
                    default='exported-models/my_mobilenet_model/saved_model/label_map.pbtxt')
parser.add_argument('--image', help='Name of the single image to perform detection on',
                    default='./img')
parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects',
                    default=0.3)
                    
args = parser.parse_args()
# Enable GPU dynamic memory allocation
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

# PROVIDE PATH TO IMAGE DIRECTORY
IMAGE_PATHS = args.image


# PROVIDE PATH TO MODEL DIRECTORY
PATH_TO_MODEL_DIR = args.model

# PROVIDE PATH TO LABEL MAP
PATH_TO_LABELS = args.labels

# PROVIDE THE MINIMUM CONFIDENCE THRESHOLD
MIN_CONF_THRESH = float(args.threshold)

# LOAD THE MODEL ==============================================================

import time
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as viz_utils

PATH_TO_SAVED_MODEL = PATH_TO_MODEL_DIR + "/saved_model"

print('Loading model...', end='')
start_time = time.time()

# LOAD SAVED MODEL AND BUILD DETECTION FUNCTION
detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)

end_time = time.time()
elapsed_time = end_time - start_time
print('Done! Took {} seconds'.format(elapsed_time))

# LOAD LABEL MAP DATA FOR PLOTTING

category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS,
                                                                    use_display_name=True)

# LOAD IMAGE ==================================================================

import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')   # Suppress Matplotlib warnings
from collections import namedtuple
import os



def load_image_into_numpy_array(path):
    """Load an image from file into a numpy array.

    Puts image into numpy array to feed into tensorflow graph.
    Note that by convention we put it into a numpy array with shape
    (height, width, channels), where channels=3 for RGB.

    Args:
      path: the file path to the image

    Returns:
      uint8 numpy array with shape (img_height, img_width, 3)
    """    
    
    return np.array(Image.open(path))

print('Running inference for {}... '.format(IMAGE_PATHS), '\n')

# ORDNER EINLESEN & BILDER ABFRAGEN ===========================================

import glob

def load_img_folder(img_folder_path):
    
    img_names = []
    img_data = []
    
    for img  in glob.glob(img_folder_path):
        
        # create image name list
        img_name = img.split('/')[-1]
        img_names.append(img_name)
        
        #create image data list
        img_data.append(cv2.imread(img))
        
    # create image name array
    file_names = np.genfromtxt(img_names ,delimiter=',', usecols=0, dtype=str)
    
    return img_names, img_data, file_names
    
# CSV DATEI EINLESEN & BILDNAMEN VERGLEICHEN ==================================

def read_csv(image_name): # ein Bild kommt hier rein
    
    csv_data = np.genfromtxt('test_labels_2.csv',delimiter=',',skip_header=1, 
                             usecols= (4, 5, 6, 7), missing_values = {0: str})
    csv_names = np.genfromtxt('test_labels_2.csv',delimiter=',', skip_header=1, 
                              usecols=0, dtype=str)
    
    truth_boxes = []

    for i in range(len(csv_names)):   # lenght = 4320
        
        if image_name in csv_names[i]:
            truth_boxes.append(csv_data[i])

    return truth_boxes

# CALCULATE IOU ===============================================================


def batch_iou(boxA, boxB):
    
    # COORDINATES OF THE INTERSECTION BOXES
    xA = max(boxA[0], boxB[0])
    yA = max(boxA[1], boxB[1])
    xB = min(boxA[2], boxB[2])
    yB = min(boxA[3], boxB[3])
   
    # compute the area of intersection rectangle
    interArea = max(0, xB - xA + 1) * max(0, yB - yA + 1)

    
    # compute the area of both the prediction and ground-truth rectangles
    boxAArea = (boxA[2] - boxA[0] + 1) * (boxA[3] - boxA[1] + 1)
    boxBArea = (boxB[2] - boxB[0] + 1) * (boxB[3] - boxB[1] + 1)
    
    # compute the intersection over union by taking the intersection
    # area and dividing it by the sum of prediction + ground-truth
    # areas - the interesection area
    iou = interArea / float(boxAArea + boxBArea - interArea)
    
    # return the intersection over union value
    return iou

def possible_iou(truthbox_list, prediction_box):
    
    possible_iou_list = []
    
    for box in truthbox_list:
        iou = batch_iou(box, prediction_box)
        possible_iou_list.append(iou)  # Liste mit möglichen iou's
                
    return possible_iou_list
# WRITE NEW CSV FILE WITH CALCULATED IOU ======================================

import csv
        
def create_new_csv(data): 
       
    with open('output.csv', 'w') as csv_file:
        
        # Überschriften erstellen
        fieldnames = ['image name', 'gt_xmin', 'gt_ymin', 'gt_xmax', 'gt_ymax', 
                      'pred_xmin', 'pred_ymin', 'pred_max', 'pred_ymax', 'iou', 'score']
        writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
        writer.writeheader()
        
        writer = csv.writer(csv_file, delimiter=',')
        writer.writerows(data)


#==============================================================================
k = 0

img_names, img_data, file_names = load_img_folder('./img/*.jpg')

data_for_csv = []
iou_img_name = []
iou_used = []
used_tboxes = []
chosen_tboxes = []
best_ious = []
pred_boxes = []

while k < len(img_names):
    
    # truth_boxes = [array([557., 505., 817., 719.]), array([ 937.,  385., 1130.,  555.]), array([760., 371., 931., 527.])]
    truth_boxes = read_csv(str(img_names[k]))
    image = img_data[k]
   
    
    #image_rgb = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    imH, imW, _ = image.shape
    #image_expanded = np.expand_dims(image_rgb, axis=0)
    
    # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
    input_tensor = tf.convert_to_tensor(image)
    # The model expects a batch of images, so add an axis with `tf.newaxis`.
    input_tensor = input_tensor[tf.newaxis, ...]
    
    # input_tensor = np.expand_dims(image_np, 0)
    detections = detect_fn(input_tensor)
    # All outputs are batches tensors.
    # Convert to numpy arrays, and take index [0] to remove the batch dimension.
    # We're only interested in the first num_detections.
    num_detections = int(detections.pop('num_detections'))
    detections = {key: value[0, :num_detections].numpy()
                   for key, value in detections.items()}
    detections['num_detections'] = num_detections
    
    # detection_classes should be ints.
    detections['detection_classes'] = detections['detection_classes'].astype(np.int64)
    scores = detections['detection_scores']
    boxes = detections['detection_boxes']
    classes = detections['detection_classes']
    count = 0
    

    for i in range(len(scores)): # für jede predicition
        if ((scores[i] > MIN_CONF_THRESH) and (scores[i] <= 1.0)):
            #increase count
            count += 1
            # Get bounding box coordinates and draw box
            # Interpreter can return coordinates that are outside of image dimensions, need to force them to be within image using max() and min()
            ymin = int(max(1,(boxes[i][0] * imH)))
            xmin = int(max(1,(boxes[i][1] * imW)))
            ymax = int(min(imH,(boxes[i][2] * imH)))
            xmax = int(min(imW,(boxes[i][3] * imW)))
            
            cv2.rectangle(image, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2)
            # Draw label
            object_name = category_index[int(classes[i])]['name'] # Look up object name from "labels" array using class index
            label = '%s: %d%%' % (object_name, int(scores[i]*100)) # Example: 'person: 72%'
            labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) # Get font size
            label_ymin = max(ymin, labelSize[1] + 10) # Make sure not to draw label too close to top of window
            cv2.rectangle(image, (xmin, label_ymin-labelSize[1]-10), (xmin+labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED) # Draw white box to put label text in
            cv2.putText(image, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2) # Draw label text
            
            pred_box = np.array([xmin, ymin, xmax, ymax])
            score = scores[i]*100
            pred_boxes.append(pred_box)
             
            possible_iou_list = possible_iou(truth_boxes, pred_box)
            
            best_iou = max(possible_iou_list) # bester iou für diese prediction errechnen
            n = possible_iou_list.index(max(possible_iou_list))  # index des besten ious
            best_ious.append(best_iou)  # beste ious aller predicition in liste 
            
            chosen_tbox = truth_boxes[n] # beste truth box pro prediction
            chosen_tboxes.append(chosentbox)  # beste truth boxen aller predictions in liste
        
            
    while len(used_tboxes) < count: # solange Länge der Liste used_tboxes < count 
                  
        for c_tbox in range(len(chosen_tboxes)): # für jeden wert für die Länge der iou_img_names (also von 0 bis count) -- eine gewählte box kommt rein
            
            for u_tbox in range(len(used_tboxes)):
                
                for p_box in range(len(pred_boxes)):
                
                    if chosen_tboxes[c_tbox] in used_tboxes[u_tbox]:
                        
                        if best_ious[c_tbox] > iou_used[u_tbox]:
                            del used_tboxes[u_tbox]
                            del iou_used[u_tbox]
                            del pred_boxes[p_box]
                            used_tboxes.append(chosen_tboxes[c_tbox])
                            iou_used.append(best_ious[c_tbox])
                            
                        else:
                            used_tboxes.append(chosen_tboxes[c_tbox])
                            iou_used.append(best_ious[c_tbox])
                            
                        
                    else:
                        used_tboxes.append(chosen_tboxes[c_tbox])
                        iou_used.append(best_ious[c_tbox])
                 
                    
                    
                    
                if chosen_tboxes[c_tbox] in used_boxes[u_tbox]:

        # truth_boxes = [array([557., 505., 817., 719.]), array([ 937.,  385., 1130.,  555.]), array([760., 371., 931., 527.])]
        
        
        real_truth_box = box[]
        minx = int(real_truth_box[0])
        miny = int(real_truth_box[1])
        maxx = int(real_truth_box[2])
        maxy = int(real_truth_box[3])
        cv2.rectangle(image, (minx,miny), (maxx,maxy), (255, 0, 0), 2)
                
                
                
#                for i in range(len(csv_names)):   # lenght = 4320
#        
#        if image_name in csv_names[i]:
#            truth_boxes.append(csv_data[i])
                
                
            iou = batch_iou(real_truth_box, pred_box)       
            
            print('Image:', img_names[k])                
            print('Predictionbox:', pred_box, ' ', 'Ground truth box:', real_truth_box, '\033[1m' +'\nIoU=', iou, '\033[0m')
           
            
            data = [str(img_names[k]), minx, miny, maxx, maxy, xmin, ymin, xmax, ymax, iou, score]
            data_for_csv.append(data)
            # [['CARDS_LIVINGROOM_H_S_frame_2149.jpg', '[728.0, 470.0, 1008.0, 719.0]', '[ 738  476 1016  717]', '0.9085141163106419'], 
            # ['CARDS_LIVINGROOM_H_S_frame_2149.jpg', '[674.0, 400.0, 847.0, 510.0]', '[674 403 874 501]', '0.7834629553827261'], ...
            #print(data_for_csv)
            
            
        cv2.putText(image, "IoU: {:.4f}".format(iou), (xmax, label_ymin-7),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 0, 0), 2)
    k += 1
          

    cv2.putText (image,'Total Detections : ' + str(count),(10,25),cv2.FONT_HERSHEY_SIMPLEX,1,(70,235,52),2,cv2.LINE_AA)
    create_new_csv(data_for_csv)
    print('\nDone')
    
    # display output image and destroy after 5 seconds
    cv2.imshow('Object Counter', image)
    cv2.waitKey(0) 
    cv2.destroyWindow('Object Counter')

    
       
# CLOSES WINDOW ONCE KEY IS PRESSED
cv2.waitKey(0)
# CLEANUP
cv2.destroyAllWindows()

<file_sep>#!/usr/bin/env python
# coding: utf-8
"""
Object Detection (On Image) From TF2 Saved Model
=====================================
"""

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'    # Suppress TensorFlow logging (1)
import pathlib
import tensorflow as tf
import cv2
import argparse

tf.get_logger().setLevel('ERROR')           # Suppress TensorFlow logging (2)

parser = argparse.ArgumentParser()
parser.add_argument('--model', help='Folder that the Saved Model is Located In',
                    default='exported-models/my_mobilenet_model')
parser.add_argument('--labels', help='Where the Labelmap is Located',
                    default='exported-models/my_mobilenet_model/saved_model/label_map.pbtxt')
parser.add_argument('--image', help='Name of the single image to perform detection on',
                    default='./img')
parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects',
                    default=0.3)
                    
args = parser.parse_args()
# Enable GPU dynamic memory allocation
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

# PROVIDE PATH TO IMAGE DIRECTORY
IMAGE_PATHS = args.image


# PROVIDE PATH TO MODEL DIRECTORY
PATH_TO_MODEL_DIR = args.model

# PROVIDE PATH TO LABEL MAP
PATH_TO_LABELS = args.labels

# PROVIDE THE MINIMUM CONFIDENCE THRESHOLD
MIN_CONF_THRESH = float(args.threshold)

# LOAD THE MODEL ==============================================================

import time
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as viz_utils

PATH_TO_SAVED_MODEL = PATH_TO_MODEL_DIR + "/saved_model"

print('Loading model...', end='')
start_time = time.time()

# LOAD SAVED MODEL AND BUILD DETECTION FUNCTION
detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)

end_time = time.time()
elapsed_time = end_time - start_time
print('Done! Took {} seconds'.format(elapsed_time))

# LOAD LABEL MAP DATA FOR PLOTTING

category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS,
                                                                    use_display_name=True)

# LOAD IMAGE ==================================================================

import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')   # Suppress Matplotlib warnings
from collections import namedtuple
import os




def load_image_into_numpy_array(path):
    """Load an image from file into a numpy array.

    Puts image into numpy array to feed into tensorflow graph.
    Note that by convention we put it into a numpy array with shape
    (height, width, channels), where channels=3 for RGB.

    Args:
      path: the file path to the image

    Returns:
      uint8 numpy array with shape (img_height, img_width, 3)
    """    
    
    return np.array(Image.open(path))

print('Running inference for {}... '.format(IMAGE_PATHS), '\n')

# ORDNER EINLESEN & BILDER ABFRAGEN ===========================================

import glob

def load_img_folder(img_folder_path):
    
    img_names = []
    img_data = []
    
    for img  in glob.glob(img_folder_path):
        
        # create image name list
        img_name = img.split('/')[-1]
        img_names.append(img_name)
        
        #create image data list
        img_data.append(cv2.imread(img))
        
    # create image name array
    file_names = np.genfromtxt(img_names ,delimiter=',', usecols=0, dtype=str)
    
    return img_names, img_data, file_names
    
# CSV DATEI EINLESEN & BILDNAMEN VERGLEICHEN ==================================

def read_csv(image_name): # ein Bild kommt hier rein
    
    csv_data = np.genfromtxt('test_labels_2.csv',delimiter=',',skip_header=1, 
                             usecols= (4, 5, 6, 7), missing_values = {0: str})
    csv_names = np.genfromtxt('test_labels_2.csv',delimiter=',', skip_header=1, 
                              usecols=0, dtype=str)
    
    truth_boxes = []

    for i in range(len(csv_names)):   # lenght = 4320
        
        if image_name in csv_names[i]:
            truth_boxes.append(csv_data[i])

    return truth_boxes

# CALCULATE IOU ===============================================================


def batch_iou(boxA, boxB):
    
    # COORDINATES OF THE INTERSECTION BOXES
    xA = max(boxA[0], boxB[0])
    yA = max(boxA[1], boxB[1])
    xB = min(boxA[2], boxB[2])
    yB = min(boxA[3], boxB[3])
   
    # compute the area of intersection rectangle
    interArea = max(0, xB - xA + 1) * max(0, yB - yA + 1)

    
    # compute the area of both the prediction and ground-truth rectangles
    boxAArea = (boxA[2] - boxA[0] + 1) * (boxA[3] - boxA[1] + 1)
    boxBArea = (boxB[2] - boxB[0] + 1) * (boxB[3] - boxB[1] + 1)
    
    # compute the intersection over union by taking the intersection
    # area and dividing it by the sum of prediction + ground-truth
    # areas - the interesection area
    iou = interArea / float(boxAArea + boxBArea - interArea)
    
    # return the intersection over union value
    return iou

import scipy.optimize

def match_bboxes(bbox_gt, bbox_pred, IOU_THRESH=0.5):
    '''
    Given sets of true and predicted bounding-boxes,
    determine the best possible match.
    Parameters
    ----------
    bbox_gt, bbox_pred : N1x4 and N2x4 np array of bboxes [x1,y1,x2,y2]. 
      The number of bboxes, N1 and N2, need not be the same.
    
    Returns
    -------
    (idxs_true, idxs_pred, ious, labels)
        idxs_true, idxs_pred : indices into gt and pred for matches
        ious : corresponding IOU value of each match
        labels: vector of 0/1 values for the list of detections
    '''
    n_true = bbox_gt.shape[0] # Anzahl der truth boxes
    n_pred = bbox_pred.shape[0] # Anzahl der prediction boxen
    MAX_DIST = 1.0
    MIN_IOU = 0.0

    # NUM_GT x NUM_PRED
    iou_matrix = np.zeros((n_true, n_pred))
    for i in range(n_true): # für jede truth box
        for j in range(n_pred): # für jede pred box
            iou_matrix[i, j] = batch_iou(bbox_gt[i,:], bbox_pred[j,:]) # iou berechnen aus einzelner gt box und pred box
    
    if n_pred > n_true:
      # there are more predictions than ground-truth - add dummy rows
      diff = n_pred - n_true
      print("hier sind mehr predictions als gt")
      iou_matrix = np.concatenate( (iou_matrix, 
                                    np.full((diff, n_pred), MIN_IOU)), 
                                  axis=0)
      
    if n_true > n_pred:
      # more ground-truth than predictions - add dummy columns
      diff = n_true - n_pred
      #print("hier sind mehr gt als pred")
      iou_matrix = np.concatenate( (iou_matrix, 
                                    np.full((n_true, diff), MIN_IOU)), 
                                  axis=1)
    #print('2', iou_matrix)
    # call the Hungarian matching
    idxs_true, idxs_pred = scipy.optimize.linear_sum_assignment(1 - iou_matrix) # truth boxen und pred box arrays in gleicher Reihenfolge

    if (not idxs_true.size) or (not idxs_pred.size):
        ious = np.array([])
    else:
        ious = iou_matrix[idxs_true, idxs_pred] # ious in der selben reihenfolge wie truth boxen und pred boxen im array

    # remove dummy assignments
    sel_pred = idxs_pred<n_pred  # hat das array der neuen reihenfolge eine kleinere anzahl an elemenzen als n_pred? [True True True] für 3 elemente in reihenfolge
    idx_pred_actual = idxs_pred[sel_pred] # aktuelle neue reihenfole der pred boxen speichern
    print(idx_pred_actual)
    idx_gt_actual = idxs_true[sel_pred] # aktuelle neue reihenfole der gt boxen speichern
    print(idx_gt_actual)
    ious_actual = iou_matrix[idx_gt_actual, idx_pred_actual] # aklteulle ious in array nach neuer reihenfolge gespeichter (wenn doppelte =  0)
    print(ious_actual)
    sel_valid = (ious_actual > IOU_THRESH) # wenn iou=0 dann false, sonst true in array
    label = sel_valid.astype(int) # true = 1, false = 0

    return idx_gt_actual[sel_valid], idx_pred_actual[sel_valid], ious_actual[sel_valid], label 

# WRITE NEW CSV FILE WITH CALCULATED IOU ======================================

import csv
#from itertools import zip_longest
import operator
from functools import reduce
        
def create_new_csv(data): 
       
    with open('output.csv', 'w') as csv_file:
        
        # Überschriften erstellen
        fieldnames = ['image name', 'gt_xmin', 'gt_ymin', 'gt_xmax', 'gt_ymax', 
                      'pred_xmin', 'pred_ymin', 'pred_max', 'pred_ymax', 'iou',
                      'score', 'true positive = 1, false positive = 0']
        writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
        writer.writeheader()
        
        writer = csv.writer(csv_file, delimiter=',')
        
        #d = [data, true_positives]
        #export_data = zip_longest(*d, fillvalue = '')
        #writer.writerows(zip(data, true_positives))
        #print(true_positives)
        
        writer.writerows(data)
        #writer.writerows(positives_data)
        #writer.writerows(export_data)

#=======================================bboxes=======================================
k = 0

img_names, img_data, file_names = load_img_folder('./img/*.jpg')

data_for_csv = []
pred_boxes = []
true_positives = []
false_positives = []


while k < len(img_names):
    
    # truth_boxes = [array([557., 505., 817., 719.]), array([ 937.,  385., 1130.,  555.]), array([760., 371., 931., 527.])]
    truth_boxes = read_csv(str(img_names[k]))
    image = img_data[k]
   
    
    #image_rgb = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    imH, imW, _ = image.shape
    #image_expanded = np.expand_dims(image_rgb, axis=0)
    
    # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
    input_tensor = tf.convert_to_tensor(image)
    # The model expects a batch of images, so add an axis with `tf.newaxis`.
    input_tensor = input_tensor[tf.newaxis, ...]
    
    # input_tensor = np.expand_dims(image_np, 0)
    detections = detect_fn(input_tensor)
    # All outputs are batches tensors.
    # Convert to numpy arrays, and take index [0] to remove the batch dimension.
    # We're only interested in the first num_detections.
    num_detections = int(detections.pop('num_detections'))
    detections = {key: value[0, :num_detections].numpy()
                   for key, value in detections.items()}
    detections['num_detections'] = num_detections
    
    # detection_classes should be ints.
    detections['detection_classes'] = detections['detection_classes'].astype(np.int64)
    scores = detections['detection_scores']
    boxes = detections['detection_boxes']
    classes = detections['detection_classes']
    count = 0
    

    for i in range(len(scores)):
        if ((scores[i] > MIN_CONF_THRESH) and (scores[i] <= 1.0)):
            #increase count
            count += 1
            # Get bounding box coordinates and draw box
            # Interpreter can return coordinates that are outside of image dimensions, need to force them to be within image using max() and min()
            ymin = int(max(1,(boxes[i][0] * imH)))
            xmin = int(max(1,(boxes[i][1] * imW)))
            ymax = int(min(imH,(boxes[i][2] * imH)))
            xmax = int(min(imW,(boxes[i][3] * imW)))
            
            cv2.rectangle(image, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2)
            # Draw label
            object_name = category_index[int(classes[i])]['name'] # Look up object name from "labels" array using class index
            label = '%s: %d%%' % (object_name, int(scores[i]*100)) # Example: 'person: 72%'
            labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) # Get font size
            label_ymin = max(ymin, labelSize[1] + 10) # Make sure not to draw label too close to top of window
            #cv2.rectangle(image, (xmin, label_ymin-labelSize[1]-10), (xmin+labelSize[0], label_ymin+baseLine-10), (255, 255, 255), cv2.FILLED) # Draw white box to put label text in
            #cv2.putText(image, label, (xmin, label_ymin-7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2) # Draw label text
            
            pred_box = np.array([xmin, ymin, xmax, ymax])
            score = scores[i]*100

            pred_boxes.append(pred_box)
            
            for box in truth_boxes:
                
                iou = batch_iou(box, pred_box)
                
                if (iou > 0.5):
                    real_truth_box = box
                    minx = int(real_truth_box[0])
                    miny = int(real_truth_box[1])
                    maxx = int(real_truth_box[2])
                    maxy = int(real_truth_box[3])
                    cv2.rectangle(image, (minx,miny), (maxx,maxy), (255, 0, 0), 2)
                
            iou = batch_iou(real_truth_box, pred_box)       
            
            print('Image:', img_names[k])                
            print('Predictionbox:', pred_box, ' ', 'Ground truth box:', real_truth_box, '\033[1m' +'\nIoU=', iou, '\033[0m')
           
            
            data = [str(img_names[k]), minx, miny, maxx, maxy, xmin, ymin, xmax, ymax, iou, score]
            data_for_csv.append(data)
            #print(data_for_csv)            
#            [['CARDS_LIVINGROOM_H_S_frame_2149.jpg', 728, 470, 1008, 719, 738, 476, 1016, 717, 0.9085141163106419, 98.49727153778076], 
#             ['CARDS_LIVINGROOM_H_S_frame_2149.jpg', 674, 400, 847, 510, 674, 403, 874, 501, 0.7834629553827261, 95.0812578201294], ...
            
        #cv2.putText(image, "IoU: {:.4f}".format(iou), (xmax, label_ymin-7),
        #            cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 0, 0), 2)
        
    
    k += 1
    
    
    truth_boxes_arr = np.array(truth_boxes)
    print(truth_boxes_arr)
    pred_boxes_arr = np.array(pred_boxes)
    print(pred_boxes_arr)
    
    # hier kommen nur die Werte an, die true sind, also größer als 0 (außer bei Label, da sind alle drin [bsp: 1 1 0])
    idx_gt_actual, idx_pred_actual, ious_actual, label = match_bboxes(truth_boxes_arr, pred_boxes_arr, IOU_THRESH=0.5)  
    print("2:" 'Predictionbox:', idx_pred_actual, ' ', 'Ground truth box:', idx_gt_actual, '\033[1m' +'\nIoU=', ious_actual, '\033[0m')
    print("in While:", idx_pred_actual)
    print(idx_gt_actual, ious_actual, label)
    label_list = label.tolist()
    true_positives.append(label_list)
    #print(true_positives)
    
#    if np.count_nonzero(label==0) > 0: # wenn die anzahl der nullen > als 0 sind
#        zero_place = np.where(label == 0)[0] # an welchen stellen im array sind die nullen?
#        
#        print(zero_place)
   
    
    cv2.putText (image,'Total Detections : ' + str(count),(10,25),cv2.FONT_HERSHEY_SIMPLEX,1,(70,235,52),2,cv2.LINE_AA)
    
    print('\nDone')
    
    pred_boxes.clear()
    # display output image and destroy after 5 seconds
    cv2.imshow('Object Counter', image)
    cv2.waitKey(0) 
    cv2.destroyWindow('Object Counter')


true_positives_list = reduce(operator.concat, true_positives)
print(true_positives_list)

for o in range(len(data_for_csv)):
    data_module = data_for_csv[o]
    data_module.append(true_positives_list[o])

#print(data_for_csv)
create_new_csv(data_for_csv)  



#label_list = label.tolist()
#true_positives.append(label_list)     
#new_list = reduce(operator.concat, true_positives)
#print(new_list)
   
    
# CLOSES WINDOW ONCE KEY IS PRESSED
cv2.waitKey(0)
# CLEANUP
cv2.destroyAllWindows()

<file_sep># gesture_project

This repository shows all about my project "gesture recognition".

---

Here you can find the informations about [how to train a model with tensorflow](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md).

---

Credits: 
- https://github.com/gustavz/deeptraining_hands
- https://github.com/armaanpriyadarshan/Training-a-Custom-TensorFlow-2.X-Object-Detector
- calculate IoU:
  - http://ronny.rest/tutorials/module/localization_001/iou/
  - https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/

---

**Hand detection:**
  - datasets: [Egohands](http://vision.soic.indiana.edu/projects/egohands/), [Oxford](http://www.robots.ox.ac.uk/~vgg/data/hands/)
  - first try of object detection (tablets): 
    - [YouTube: Training a Custom Object Detector with TensorFlow 2.0 Custom Object Detection API 2020](https://www.youtube.com/watch?v=oqd54apcgGE)
    - [GitHub: Training-a-Custom-TensorFlow-2.X-Object-Detector](https://github.com/armaanpriyadarshan/Training-a-Custom-TensorFlow-2.x-Object-Detector)
<file_sep># How to train hand datasets with the SSD with Mobilenet v2

- [1. File Structure](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#1-file-structure)
  - [1.1 Model Folder from Tensorflow](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#11-model-folder-from-tensorflow)
  - [1.2 Workspace and Scripts Folders](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#12-workspace-and-scripts-folders)
- [2. Preparing Settings](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#2-preparing-settings)
  - [2.1 Software](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#21-software)
  - [2.2 Preparing Files](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#22-preparing-files)
    - [2.2.1 Create Label Map File](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#221-create-label-map-file)
    - [2.2.2 Generate .tfrecord-File from .csv-File](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#222-generate-tfrecord-file-from-csv-file)
    - [2.2.3 Change Configuration File](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#223-change-configuration-file)
- [3. Training](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#3-training)
  - [3.1 Train the Model](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#31-train-the-model)
  - [3.2 Monitoring realtime Training (optional)](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#32-monotoring-realtime-training-optional)
  - [3.3 Exporting the Interference Graph](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#33-exporting-the-inference-graph)
- [4. Evaluation](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#4-evaluation)
  - [4.1 Evaluate the Model](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#41-evaluate-the-model)
  - [4.2 Monotoring realtime Evaluation (optional)](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#42-monotoring-realtime-evaluation-optional)
- [5. Testing](https://github.com/gitkatrin/gesture_project/blob/master/train_hand_datasets.md#5-testing)
  
# 1. File Structure

## 1.1 Model Folder from Tensorflow
```
detection_folder/
|-- models                  # from tensorflow https://github.com/tensorflow/models
|   |-- ...
|   |-- research
|   |   |-- ...
|   |   |-- object_detection
|   |   |   |-- ...
|   |   |   |-- configs     # configurations for all TF2 models
|   |   |   |-- ...
|   |   |   |-- models      # all usable models
|   |   |   |-- packages    # setup files for tf1 and tf2
|   |   |   |-- predictors  # all usable predictors
|   |   |   |-- ...
|   |   |   |-- samples     # some usable configuration samples
|   |   |   |-- ...
|   |   |-- ...
|   |   |-- model_main_tf2.py
|   |   |-- README.md
|   |   `-- setup.py
|   `-- ...
```

## 1.2 Workspace and Scripts Folders
```
detection_folder/
|-- ...
|-- scripts                   # helpful scripts for data preparation (in this repository: https://github.com/gitkatrin/gesture_project/tree/master/scripts)
|-- workspace
|   `-- training_demo
|       |-- annotations
|       |   |-- label_map.pbtxt
|       |   |-- test.record
|       |   `-- train.record
|       |-- exported-models   # this is empty at the beginning of training, your new trained model will be here
|       |-- images
|       |   |-- test
|       |   |-- train
|       |-- models            # model which you use for training
|       |   `-- my_ssd_mobilenet_v2_fpnlite      # here where 
|       |       `-- my_ssd_mobilenet_v2.config   # copy .config from pre-trained model // This folder is empty at the beginning, later there will be the folders "eval" and "train" inside. In this folder will be the checkpoints as ".data-00000-of-00001" and ".index" format and the ".config" file. In eval and train folder will be the events in n ".v2" format.
|       |-- pre-trained-models                   # pre-trained model which you use for training (download it here: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md)
|       |   `-- ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8
|       |       |-- checkpoint
|       |       |-- saved_model
|       |       `-- pipeline.config              # configuration file
|       |-- exporter_main_v2.py
|       |-- model_main_tf2.py
|       |-- TF-image-object-counting.py
|       |-- TF-image-od.py
|       |-- TF-video-object-counting.py
|       |-- TF-video-od.py
|       `-- TF-webcam-opencv.py
`-- Training-a-Custom-TensorFlow-2.X-Object-Detector-master.zip
```

[comment]: # (---------------------------------------------------------------------------------------------------------------------------------------------------------------)

# 2. Preparing Settings

## 2.1 Software
- Tensorflow, Tensorflow-gpu
- Python 3
- [Tensorflow models](https://github.com/tensorflow/models)
- Python environement (```python3 -m venv --system-site-packages ./venv```)
  - ```pip3 install protobuf```
- [scripts for data preparation](https://github.com/gitkatrin/gesture_project/tree/master/scripts)
- [Pre-trained Model](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md) (in this case: SSD MobileNet V2 FPNLite 640x640)
- Datasets: [Oxford Dataset](https://www.robots.ox.ac.uk/~vgg/data/hands/), [Egohands Dataset](http://vision.soic.indiana.edu/projects/egohands/)


## 2.2 Preparing Files

### 2.2.1 Create Label Map File

1. create a file named ```label_map.pbtxt``` in this folder: ```direction_folder/workspace/training_demo/annotations```
2. open the file and identify your items that you would like to detect. It should look like this:
    ```
    item {
    id: 1
    name: 'hand'
    }
    item {
    id: 2
    name: 'face'
    }
    ```
    the file for hand detection is [here](https://github.com/gitkatrin/gesture_project/blob/master/scripts/label_map.pbtxt)

### 2.2.2 Generate .tfrecord-File from .csv-File

1. go to the research folder in models with ```cd models/reseach``` without virtual environment
2. compile the protos with:
    ```
    protoc object_detection/protos/*.proto --python_out=.
    export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
    sudo python3 setup.py install
    ```
3. open the [csv_to_tfrecord.py](https://github.com/gitkatrin/gesture_project/blob/master/scripts/csv_to_tfrecord.py) file
4. change the path directories in the main function for:
    - Line 72: your image path
    - Line 73: your csv path
    - Line 74: your output path
    - Line 75: your label map path
5. go back into the terminal and go to path ```cd workspace/training_demo``` 
6. run the script with ```python3 csv_to_tfrecord.py```

### 2.2.3 Change Configuration File

1. copy the configuration file ```detection_folder/workspace/training_demo/pre-trained-model/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/pipeline.config``` to ```detection_folder/workspace/training_demo/models/my_ssd_mobilenet_v2_fpnlite``` folder
2. open the configuration file in ```detection_folder/workspace/training_demo/models/my_ssd_mobilenet_v2_fpnlite``` folder
3. do the following changes:
    - Line 3:   change ```num_classes``` to this value of different objects you want to detect
    - Line 135: change ```batch_size``` to 4
    - Line 165: change ```fine_tune_checkpoint``` to your path (```pre-trained-models/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/checkpoint/ckpt-0```) for transfer learning
    - Line 171: change ```fine_tune_checkpoint_type``` to ```detection```
    - Line 175: change ```label_map_path``` to your path (```annotation/label_map.pbtxt```)
    - Line 177: change ```input_path``` to your path (```annotation/train.record```)
    - Line 185: change ```label_map_path``` to your path (```annotation/label_map.pbtxt```)
    - Line 189: change ```input_path``` to your path (```annotation/test.record```)

    (keep in mind: if you choose a different model, than the line numbers will be different)
    
[comment]: # (---------------------------------------------------------------------------------------------------------------------------------------------------------------)

# 3. Training

## 3.1 Train the Model

in the **virtual environment**, in file  ```cd detection_folder/workspace/training_demo``` use the following command:
```
python3 model_main_tf2.py --model_dir=models/my_ssd_mobilenet_v2_fpnlite --pipeline_config_path=models/my_ssd_mobilenet_v2_fpnlite/my_ssd_mobilenet_v2.config
```
At the end the **loss should be between 0.150 and 0.200** to prevents unnderfitting and overfitting.


## 3.2 Monotoring realtime Training (optional)

It is possible to show the traning process with TensorBoard. This is optional.

in the **virtual environment**, in file  ```cd detection_folder/workspace/training_demo``` use the following command:
```
tensorboard --logdir=models/my_ssd_mobilenet_v2_fpnlite
```
It should open an URL-Link to the TensorBoard Server. Open this link in your web browser and you can continuously monitor training.


## 3.3 Exporting the Inference Graph

in the **virtual environment**, in file  ```cd detection_folder/workspace/training_demo``` use the following command:
```
python3 ./exporter_main_v2.py --input_type image_tensor --pipeline_config_path ./models/my_ssd_mobilenet_v2_fpnlite/my_ssd_mobilenet_v2.config --trained_checkpoint_dir ./models/my_ssd_mobilenet_v2_fpnlite/ --output_directory ./exported-models/my_mobilenet_model
```

[comment]: # (---------------------------------------------------------------------------------------------------------------------------------------------------------------)

# 4. Evaluation

## 4.1 Evaluate the Model

in the **virtual environment**, in file  ```cd detection_folder/workspace/training_demo``` use the following command:
```
python3 model_main_tf2.py --pipeline_config_path models/my_ssd_mobilenet_v2_fpnlite/my_ssd_mobilenet_v2.config --model_dir models/my_ssd_mobilenet_v2_fpnlite --checkpoint_dir models/my_ssd_mobilenet_v2_fpnlite --alsologtostderr
```

## 4.2 Monotoring realtime Evaluation (optional)

It is possible to show the evaluation process with TensorBoard. This is optional.

in the **virtual environment**, in file  ```cd detection_folder/workspace/training_demo``` use the following command:
```
tensorboard --logdir=models/my_ssd_mobilenet_v2_fpnlite
```
It should open an URL-Link to the TensorBoard Server. Open this link in your web browser and you can continuously monitor evaluation.

[comment]: # (---------------------------------------------------------------------------------------------------------------------------------------------------------------)

# 5. Testing

First you need to copy your ```label_map.pbtxt``` from ```cd detection_folder/workspace/training_demo/annotations``` to your  ```cd detection_folder/workspace/training_demo/exported-models/my_mobilenet_model/saved_model``` folder

in the **virtual environment**, in file  ```cd detection_folder/workspace/training_demo``` use the following command:
```
python3 TF-image-od.py
```
There are also some other scripts for detection with different input:
- for detecting and coundting objects on an **image**: ```python3 TF-image-object-counting.py```
- for detecting objects in a **video**: ```python3 TF-video-od.py```
- for detecting and counting objects in a **video**: ```python3 TF-video-object-counting.py```
- for detecting objects live on **webcam**: ```python3 TF-webcam-opencv.py```
- for detecting objects live on **special webcam**: ```python3 TF-webcam-OV580.py``` 

You can also run this script with different arguments:
```
python3 TF-image-od.py [-h] [--model MODEL] [--labels LABELS] [--image IMAGE] [--threshold THRESHOLD]

optional arguments:
  -h, --help            show this help message and exit
  --model MODEL         Folder that the Saved Model is Located In
  --labels LABELS       Where the Labelmap is Located
  --image IMAGE         Name of the single image to perform detection on
  --threshold THRESHOLD Minimum confidence threshold for displaying detected objects
```
