<file_sep># Gesture Detection and Recognition in TV News Videos

Communication using body language is an ancient art form, currently evolving in many fascinating ways. And automatic detection of human body language is becoming an active subject of research due to its application in various vision-based articulated body pose estimation systems such as Markerless motion capture for human-computer interfaces, Robot control, visual surveillance, Human image synthesis. A specific part of this field, gesture recognition, has gained great attention in recent years. Current focuses in the field include emotion recognition from face and hand gesture recognition. Users can use simple gestures to control or interact with devices without physically touching them. Gesture recognition has also benefited in the field of Defence, Home Automation, Automated sign language translation.

*This is my [Google Summer of Code 2019](https://summerofcode.withgoogle.com/projects/#4821012401618944) Project with [the Distributed Little Red Hen Lab](http://www.redhenlab.org).*

The aim of this project is to develop an automated system for hand gesture detection and recognition in TV news videos. Given a news video of certain time duration, the automated system should not only detect the hand gestures in the video but also provide a label from among the set of hand-gesture classes. Finally, the system should be incorporated into singularity container for deployment on high-performance clusters (HPC) clusters.

### Tools and Libraries

- openCV
- Tensorflow
- Keras
- Singularity Container

### Project Update

The progress made in the project is updated in the blog posts. The blog posts with the updates on the work done are:

- [Setting up OpenPose in CWRU HPC](https://medium.com/@abhinavpatel2912/setting-up-openpose-in-cwru-hpc-8955f510f6ac)
- [Week 1 & Week 2 (GSoC 2019)](https://medium.com/@abhinavpatel2912/week-1-week-2-gsoc-2019-85448409dd0f)
- [Week 3 & Week 4 (GSoC 2019)](https://medium.com/@abhinavpatel2912/week-3-week-4-gsoc-2019-12d572b2cd5c)
- [Week 5 (GSoC 2019)](https://medium.com/@abhinavpatel2912/week-5-gsoc-2019-6b07acc91eb6)
- [Weeks 6-8 (GSoC 2019)](https://medium.com/@abhinavpatel2912/weeks-6-8-gsoc-2019-1b3d7df33e0c)
<file_sep># Importing Libraries

import sys
import json
import os
import numpy as np

# Default video frame dimensions

img_height = 240
img_width = 320

offset = 25  # amount of shift in extreme points (in pixels)


def getExtrmCoords(points_list, img_height, img_width):
    '''

    :param points_list: list of x and y coordinates of joints along with their confidence scores
    :return: extreme coordinates in x and y directions

    '''

    x_coord = []
    y_coord = []
    #     print("points_list: ", points_list)
    for ptr in range(0, len(points_list), 3):

        curr_x = points_list[ptr]
        curr_y = points_list[ptr + 1]
        prob = points_list[ptr + 2]

        if curr_x > 0 and curr_x < img_width and prob > 0.2:
            x_coord.append(curr_x)
        if curr_y > 0 and curr_y < img_height and prob > 0.2:
            y_coord.append(curr_y)

    #     print("x_coord: ", x_coord)
    #     print("y_coord: ", y_coord)
    points_min_x = 1000
    points_max_x = 0
    points_min_y = 1000
    if len(x_coord) > 0 and len(y_coord) > 0:
        points_min_x = min(i for i in x_coord if i > 0)
        points_max_x = max(i for i in x_coord if i > 0)

        points_min_y = min(i for i in y_coord if i > 0)

    return points_min_x, points_max_x, points_min_y


def getBoundingBox(json_path, global_val, img_height, img_width):
    '''
    :param json_path: path of directory containing all the json files of the respective video
    :param global_val: list of variables storing extreme coordinates
    :return: bounding box around the person in the given video
    '''

    global_min_x, global_max_x, global_min_y, global_max_y = global_val
    skipped = 0

    file_list = os.listdir(json_path)

    for json_file in file_list:

        arr = json_file.split('.')
        if arr[-1] != "json":
            continue

        #         print("{} json started!".format(json_file))

        json_file_path = os.path.join(json_path, json_file)

        with open(json_file_path, 'r') as myfile:
            data = myfile.read()

        try:
            obj = json.loads(data)
            main_dict = obj['people'][0]
        except:
            continue

        #         print("main_dict.keys() is {}".format(main_dict.keys()))
        pose = main_dict['pose_keypoints']
        hand_left = main_dict['hand_left_keypoints']
        hand_right = main_dict['hand_right_keypoints']

        # face = main_dict['face_keypoints']

        pose_min_x, pose_max_x, pose_min_y = getExtrmCoords(pose, img_height, img_width)
        hand_left_min_x, hand_left_max_x, hand_left_min_y = getExtrmCoords(hand_left, img_height, img_width)
        hand_right_min_x, hand_right_max_x, hand_right_min_y = getExtrmCoords(hand_right, img_height, img_width)
        # face_min_x, face_max_x, face_min_y = getExtrmCoords(face)

        global_min_x = min(global_min_x, pose_min_x, hand_left_min_x, hand_right_min_x)
        global_max_x = max(global_max_x, pose_max_x, hand_left_max_x, hand_right_max_x)

        global_min_y = min(global_min_y, pose_min_y, hand_left_min_y, hand_right_min_y)

    #     print("global_min_x is {} and global_max_x is {}".format(global_min_x, global_max_x))

    if global_min_x - offset <= 0:
        global_min_x = 0
    else:
        global_min_x -= offset

    if global_max_x + offset > img_width:
        global_max_x = img_width
    else:
        global_max_x += offset

    if global_min_y - offset <= 0:
        global_min_y = 0
    else:
        global_min_y -= offset

    bounding_box = [
        global_min_x,
        global_max_x,
        global_min_y,
        global_max_y
    ]

    return bounding_box


def main(global_val):
    '''

    :param global_val: list of variables storing extreme coordinates
    :return: bounding box for each of the video of ChaLearn Dataset
    '''

    store_json = "/home/axp798/axp798gallinahome/store/json/"

    dir_list = os.listdir(store_json)
    for dir_name in dir_list:
        video_list = os.listdir(os.path.join(store_json, dir_name))

        for video_name in video_list:
            json_path = os.path.join(store_json, "{}/{}/".format(dir_name, video_name))

            bounding_box = getBoundingBox(json_path, global_val)

            return bounding_box


if __name__ == '__main__':

    global_min_x = 100000
    global_max_x = 0

    global_min_y = 0
    global_max_y = img_height

    global_val = [global_min_x, global_max_x, global_min_y, global_max_y]

    main(global_val)


<file_sep>import tensorflow as tf
import numpy as np
import os


def getKernel(name, shape):
    kernel = tf.get_variable(name, shape, tf.float32, tf.contrib.layers.xavier_initializer(uniform=True,
                                                                                           seed=None,
                                                                                           dtype=tf.float32))

    return kernel


def _bias_variable(name, shape):
    return tf.get_variable(name, shape, tf.float32, tf.constant_initializer(0.0, dtype=tf.float32))


# block 1

def conv3DBlock(prev_layer, layer_name, in_filters, out_filters):
    with tf.variable_scope(layer_name):
        kernel_shape = 3
        kernel_name = "weights"
        conv_stride = [1, 1, 1, 1, 1]

        kernel = getKernel(kernel_name, [kernel_shape, kernel_shape, kernel_shape, in_filters, out_filters])
        prev_layer = tf.nn.conv3d(prev_layer, kernel, strides=conv_stride, padding="SAME")
        prev_layer = tf.layers.batch_normalization(prev_layer, training=True)
        biases = _bias_variable('biases', [out_filters])
        prev_layer = tf.nn.bias_add(prev_layer, biases)
        prev_layer = tf.nn.relu(prev_layer)

    return prev_layer


def maxPool3DBlock(prev_layer, ksize, pool_stride):
    prev_layer = tf.nn.max_pool3d(prev_layer, ksize, pool_stride, padding="VALID")

    return prev_layer


def inference(video):
    shapes_list = []
    num_classes = 249

    prev_layer = video
    in_filters = 3

    # block 1
    out_filters = 32
    layer_name = "conv1a"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    prev_layer = maxPool3DBlock(prev_layer, [1, 1, 2, 2, 1], [1, 1, 2, 2, 1])
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    # block 2
    out_filters = 64
    layer_name = "conv2a"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    prev_layer = maxPool3DBlock(prev_layer, [1, 2, 2, 2, 1], [1, 2, 2, 2, 1])
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    # block 3
    out_filters = 128
    layer_name = "conv3a"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    out_filters = 128
    layer_name = "conv3b"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    prev_layer = maxPool3DBlock(prev_layer, [1, 2, 2, 2, 1], [1, 2, 2, 2, 1])
    shapes_list.append(prev_layer.get_shape())

    # block 4
    out_filters = 256
    layer_name = "conv4a"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    out_filters = 256
    layer_name = "conv4b"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    prev_layer = maxPool3DBlock(prev_layer, [1, 2, 2, 2, 1], [1, 2, 2, 2, 1])
    shapes_list.append(prev_layer.get_shape())

    # block 5
    out_filters = 256
    layer_name = "conv5a"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    out_filters = 256
    layer_name = "conv5b"
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    in_filters = out_filters
    print(prev_layer.get_shape())
    shapes_list.append(prev_layer.get_shape())

    prev_layer = maxPool3DBlock(prev_layer, [1, 2, 2, 2, 1], [1, 2, 2, 2, 1])
    shapes_list.append(prev_layer.get_shape())

    with tf.variable_scope('fc1'):
        dim = np.prod(prev_layer.get_shape().as_list()[1:])
        prev_layer_flat = tf.reshape(prev_layer, [-1, dim])
        weights = getKernel('weights', [dim, 512])
        biases = _bias_variable('biases', [512])
        prev_layer = tf.nn.relu(tf.matmul(prev_layer_flat, weights) + biases)
        print(prev_layer.get_shape())
        shapes_list.append(prev_layer.get_shape())

    with tf.variable_scope('dropout1'):
        prev_layer = tf.nn.dropout(prev_layer, 0.4)

    with tf.variable_scope('fc2'):
        dim = np.prod(prev_layer.get_shape().as_list()[1:])
        prev_layer_flat = tf.reshape(prev_layer, [-1, dim])
        weights = getKernel('weights', [dim, 512])
        biases = _bias_variable('biases', [512])
        prev_layer = tf.nn.relu(tf.matmul(prev_layer_flat, weights) + biases)
        print(prev_layer.get_shape())
        shapes_list.append(prev_layer.get_shape())

    with tf.variable_scope('dropout2'):
        prev_layer = tf.nn.dropout(prev_layer, 0.4)

    with tf.variable_scope('Softmax_Layer'):
        dim = np.prod(prev_layer.get_shape().as_list()[1:])
        prev_layer_flat = tf.reshape(prev_layer, [-1, dim])
        weights = getKernel('weights', [dim, num_classes])
        biases = _bias_variable('biases', [num_classes])
        softmax_linear = tf.add(tf.matmul(prev_layer_flat, weights), biases)
        print(softmax_linear.get_shape())
        shapes_list.append(prev_layer.get_shape())

    return softmax_linear, shapes_list
<file_sep>import os
import numpy as np
import tensorflow as tf
from tensorflow.python.client import device_lib
import time
# import scipy.misc as smisc
import pickle
import random


def getKernel(name, shape):
    kernel = tf.get_variable(name, shape, tf.float32, tf.contrib.layers.xavier_initializer(uniform=True,
                                                                                           seed=None,
                                                                                           dtype=tf.float32))
    return kernel


def _bias_variable(name, shape):
    return tf.get_variable(name, shape, tf.float32, tf.constant_initializer(0.0, dtype=tf.float32))


def maxPool3DBlock(prev_layer, ksize, pool_stride):
    prev_layer = tf.nn.max_pool3d(prev_layer, ksize, pool_stride, padding="VALID")

    return prev_layer


def conv3DBlock(prev_layer, layer_name, in_filters, out_filters):
    kernel_shape = 3
    kernel_name = "weights"
    conv_stride = [1, 1, 1, 1, 1]
    with tf.variable_scope(layer_name):
        kernel = getKernel(kernel_name, [kernel_shape, kernel_shape, kernel_shape, in_filters, out_filters])
        prev_layer = tf.nn.conv3d(prev_layer, kernel, strides=conv_stride, padding="SAME")
        # print(layer_name, prev_layer.get_shape())

    return prev_layer


def inpuT_block(inpuT, inpuT_filter_shape, inpuT_stride):
    with tf.variable_scope("inpuT_layer"):
        out_filters = 3
        kernel_name = "weights"
        kernel = getKernel(kernel_name, inpuT_filter_shape)
        curr_layer = tf.nn.conv3d(inpuT, kernel, strides=inpuT_stride, padding="SAME")
        curr_layer = tf.layers.batch_normalization(curr_layer, training=True)
        biases = _bias_variable('biases', [out_filters])
        curr_layer = tf.nn.bias_add(curr_layer, biases)
        curr_layer = tf.nn.relu(curr_layer)

    return curr_layer


def basic_block(inpuT, in_filters, out_filters, layer_num, add_kernel_name):
    layer_name = "conv{}a".format(layer_num)
    prev_layer = conv3DBlock(inpuT, layer_name, in_filters, out_filters)
    prev_layer = tf.layers.batch_normalization(prev_layer, training=True)
    prev_layer = tf.nn.relu(prev_layer)

    in_filters = out_filters

    layer_name = "conv{}b".format(layer_num)
    prev_layer = conv3DBlock(prev_layer, layer_name, in_filters, out_filters)
    prev_layer = tf.layers.batch_normalization(prev_layer, training=True)

    residual = prev_layer

    prev_layer = _shortcut3d(inpuT, residual, add_kernel_name)
    prev_layer = tf.nn.relu(prev_layer)

    return prev_layer


def _shortcut3d(inpuT, residual, add_kernel_name):
    """3D shortcut to match inpuT and residual and sum them."""
    conv_stride = [1, 1, 1, 1, 1]
    inpuT_shape = inpuT.get_shape().as_list()
    residual_shape = residual.get_shape().as_list()

    equal_channels = inpuT_shape[-1] == residual_shape[-1]
    shortcut = inpuT

    if not equal_channels:
        kernel = getKernel(add_kernel_name, [1, 1, 1, inpuT_shape[-1], residual_shape[-1]])
        shortcut = tf.nn.conv3d(shortcut, kernel, strides=conv_stride, padding="SAME")

    return (shortcut + residual)


def inference(inpuT):
    num_filters_list = [16, 32, 64, 128]  # as per the paper

    num_classes = 27
    num_channels = 3
    inpuT_filter_shape = [3, 7, 7, num_channels, num_channels]
    inpuT_stride = [1, 1, 2, 2, 1]

    print (inpuT.get_shape())

    prev_layer = inpuT_block(inpuT, inpuT_filter_shape, inpuT_stride)
    print(prev_layer.get_shape())
    in_filters = 3

    for index, out_filters in enumerate(num_filters_list):
        add_kernel_name = "add{}".format(index + 1)
        prev_layer = basic_block(prev_layer, in_filters, out_filters, index + 2, add_kernel_name)
        print(prev_layer.get_shape())

        if index > 0:
            ksize = [1, 2, 1, 1, 1]
            pool_stride = [1, 2, 1, 1, 1]
            prev_layer = maxPool3DBlock(prev_layer, ksize, pool_stride)

        print(prev_layer.get_shape())
        in_filters = out_filters

    shape_list = prev_layer.get_shape().as_list()

    num_frames = shape_list[1]
    height = shape_list[2]
    width = shape_list[3]

    prev_layer = tf.nn.avg_pool3d(prev_layer, [1, num_frames, height, width, 1], [1, 1, 1, 1, 1], padding="VALID")
    print(prev_layer.get_shape())

    with tf.variable_scope('fc1') as scope:
        dim = np.prod(prev_layer.get_shape().as_list()[1:])
        prev_layer_flat = tf.reshape(prev_layer, [-1, dim])
        weights = getKernel('weights', [dim, 64])
        biases = _bias_variable('biases', [64])

        prev_layer = tf.nn.relu(tf.matmul(prev_layer_flat, weights) + biases, name=scope.name)
        print(prev_layer.get_shape())

    #
    # with tf.variable_scope('dropout1'):
    #     prev_layer = tf.nn.dropout(prev_layer, 0.4)

    with tf.variable_scope('fc2') as scope:
        dim = np.prod(prev_layer.get_shape().as_list()[1:])
        prev_layer_flat = tf.reshape(prev_layer, [-1, dim])
        weights = getKernel('weights', [dim, 64])
        biases = _bias_variable('biases', [64])

        prev_layer = tf.nn.relu(tf.matmul(prev_layer_flat, weights) + biases, name=scope.name)
        print(prev_layer.get_shape())

    # with tf.variable_scope('dropout2'):
    #     prev_layer = tf.nn.dropout(prev_layer, 0.4)

    with tf.variable_scope('softmax_linear') as scope:
        dim = np.prod(prev_layer.get_shape().as_list()[1:])
        prev_layer_flat = tf.reshape(prev_layer, [-1, dim])

        weights = getKernel('weights', [dim, num_classes])
        biases = _bias_variable('biases', [num_classes])

        softmax_linear = tf.add(tf.matmul(prev_layer_flat, weights), biases, name=scope.name)

        print(softmax_linear.get_shape())

    return softmax_linear
<file_sep>import os

openpose_simg = "/home/axp798/axp798gallinahome/openpose_1_0_5.simg"  # Path to openpose singularity image being used
store_path = "/home/axp798/axp798gallinahome/"  # Path to directory for storing rendered videos and json files from train videos
json_store = os.path.join(store_path, "JD_JSON/")  # directory to store json files containing hand, face and body coordinates from the videos

jd_videos_dir = "/home/axp798/axp798gallinahome/jester_datase/"  # Path to list of directories containing training videos
lib_dir = "/home/axp798/axp798gallinahome/"  # Path to directory to be accessible from within the singularity container


def main():
    video_ptr = 0
    videos_list = os.listdir(jd_videos_dir)

    for video_name in videos_list:

        print("Video {}-{} started!".format(video_ptr + 1, video_name))
        images_dir = os.path.join(jd_videos_dir, video_name)
        modified_video_name = ('0' * (6 - len(video_name))) + video_name  ## 2 --> 000002
        print(modified_video_name)
        if not os.path.exists(os.path.join(json_store, "{}/".format(modified_video_name))):
            os.mkdir(os.path.join(json_store, "{}/".format(modified_video_name)))

        video_path = images_dir
        write_json_path = os.path.join(json_store, "{}/".format(modified_video_name))

        command = "singularity run --nv --bind {} {} --image_dir {} --write_keypoint_json {} --no_display --render_pose 0 --hand".format(
                    lib_dir,
                    openpose_simg,
                    video_path,
                    write_json_path)
        os.system(command)  # run the command on terminal
        video_ptr += 1
        print("Video {} completed!".format(video_ptr))


if __name__ == '__main__':
    main()

<file_sep>import os
import random

import numpy as np
import tensorflow as tf
from tensorflow.contrib.framework import arg_scope
from tensorflow.contrib.layers import batch_norm, flatten
from tensorflow.python.client import device_lib

weight_decay = 0.0005
momentum = 0.9

init_learning_rate = 0.001
cardinality = 8  # how many split ?
blocks = 3  # res_block ! (split + transition)

"""
So, the total number of layers is (3*blokcs)*residual_layer_num + 2
because, blocks = split(conv 2) + transition(conv 1) = 3 layer
and, first conv layer 1, last dense layer 1
thus, total number of layers = (3*blocks)*residual_layer_num + 2
"""

depth = 64  # out channel

image_size = 112
image_depth = 16
num_channels = 3
num_classes = 249


def conv_layer(input, filter, kernel, stride, padding='SAME', layer_name="conv"):
    with tf.name_scope(layer_name):
        network = tf.layers.conv3d(inputs=input, use_bias=False, filters=filter, kernel_size=kernel, strides=stride,
                                   padding=padding)
        return network


def Global_Average_Pooling(x):
    x_shape = x.get_shape().as_list()
    depth = x_shape[1]
    height = x_shape[2]
    width = x_shape[3]
    return tf.layers.average_pooling3d(inputs=x, pool_size=[depth, height, width], strides=[1, 1, 1], padding='valid')


def Max_pooling(x, pool_size=[2, 2, 2], stride=2, padding='SAME'):
    return tf.layers.max_pooling3d(inputs=x, pool_size=pool_size, strides=stride, padding=padding)


def Batch_Normalization(x, training, scope):
    with arg_scope([batch_norm],
                   scope=scope,
                   updates_collections=None,
                   decay=0.9,
                   center=True,
                   scale=True,
                   zero_debias_moving_mean=True):
        return tf.cond(training,
                       lambda: batch_norm(inputs=x, is_training=training, reuse=None),
                       lambda: batch_norm(inputs=x, is_training=training, reuse=True))


def Relu(x):
    return tf.nn.relu(x)


def Concatenation(layers):
    return tf.concat(layers, axis=4)


def Linear(x):
    return tf.layers.dense(inputs=x, use_bias=False, units=num_classes, name='linear' + str(random.random()))


def Evaluate(sess):
    test_acc = 0.0
    test_loss = 0.0
    test_pre_index = 0
    add = 1000

    for it in range(test_iteration):
        test_batch_x = test_x[test_pre_index: test_pre_index + add]
        test_batch_y = test_y[test_pre_index: test_pre_index + add]
        test_pre_index = test_pre_index + add

        test_feed_dict = {
            x: test_batch_x,
            label: test_batch_y,
            learning_rate: epoch_learning_rate,
            training_flag: False
        }

        loss_, acc_ = sess.run([cost, accuracy], feed_dict=test_feed_dict)

        test_loss += loss_
        test_acc += acc_

    test_loss /= test_iteration  # average loss
    test_acc /= test_iteration  # average accuracy

    summary = tf.Summary(value=[tf.Summary.Value(tag='test_loss', simple_value=test_loss),
                                tf.Summary.Value(tag='test_accuracy', simple_value=test_acc)])

    return test_acc, test_loss, summary


class ResNeXt(object):
    def __init__(self, x, training):
        self.training = training
        self.model = self.Build_ResNext(x)

    def first_layer(self, x, scope):
        with tf.name_scope(scope):
            x = conv_layer(x, filter=64, kernel=[7, 7, 7], stride=[1, 2, 2], padding='SAME',
                           layer_name=scope + '_conv1' + str(random.random()))
            x = Batch_Normalization(x, training=self.training,
                                    scope=scope + '_batch1{}'.format(random.randint(10, 100)))
            x = Max_pooling(x, pool_size=[3, 3, 3], stride=2, padding='SAME')
            x = Relu(x)

            return x

    def transform_layer(self, x, stride, scope):
        with tf.name_scope(scope):
            x = conv_layer(x, filter=depth, kernel=[1, 1, 1], stride=stride,
                           layer_name=scope + '_conv1' + str(random.random()))
            x = Batch_Normalization(x, training=self.training, scope=scope + '_batch1'.format(random.randint(20, 200)))
            x = Relu(x)

            x = conv_layer(x, filter=depth, kernel=[3, 3, 3], stride=1,
                           layer_name=scope + '_conv2' + str(random.random()))
            x = Batch_Normalization(x, training=self.training, scope=scope + '_batch2'.format(random.randint(30, 300)))
            x = Relu(x)
            return x

    def transition_layer(self, x, out_dim, scope):
        with tf.name_scope(scope):
            x = conv_layer(x, filter=out_dim, kernel=[1, 1, 1], stride=1,
                           layer_name=scope + '_conv1' + str(random.random()))
            x = Batch_Normalization(x, training=self.training, scope=scope + '_batch1'.format(random.randint(40, 400)))
            # x = Relu(x)

            return x

    def split_layer(self, input_x, stride, layer_name):
        with tf.name_scope(layer_name):
            layers_split = list()
            for i in range(cardinality):
                splits = self.transform_layer(input_x, stride=stride,
                                              scope=layer_name + '_splitN_' + str(i) + str(random.random()))
                layers_split.append(splits)

            return Concatenation(layers_split)

    def residual_layer(self, input_x, out_dim, layer_num, res_block=blocks):
        # split + transform(bottleneck) + transition + merge
        for i in range(res_block):
            # input_dim = input_x.get_shape().as_list()[-1]
            input_dim = int(np.shape(input_x)[-1])

            channel = None
            if input_dim * 2 == out_dim:
                flag = True
                stride = 2
                channel = int(input_dim / 2)

            else:
                flag = False
                stride = 1

            x = self.split_layer(input_x, stride=stride,
                                 layer_name='split_layer_' + layer_num + '_' + str(i) + str(random.random()))
            x = self.transition_layer(x, out_dim=out_dim,
                                      scope='trans_layer_' + layer_num + '_' + str(i) + str(random.random()))

            if flag is True:
                pad_input_x = Max_pooling(input_x)
                pad_input_x = tf.pad(pad_input_x, [[0, 0], [0, 0], [0, 0], [0, 0], [channel, channel]])
            else:

                x = conv_layer(input_x, filter=out_dim, kernel=[1, 1, 1], stride=[2, 2, 2],
                               layer_name='conv_residual' + str(random.random()))
                x = Batch_Normalization(x, training=self.training, scope='batch_residual{}'.format(random.random()))
                pad_input_x = x

            input_x = Relu(x + pad_input_x)
            print(input_x.get_shape())

        return input_x

    def Build_ResNext(self, input_x):
        # only cifar10 architecture
        input_x = self.first_layer(input_x, scope='first_layer')
        print(input_x.get_shape())
        x = self.residual_layer(input_x, out_dim=256, layer_num='1', res_block=3)
        x = self.residual_layer(x, out_dim=512, layer_num='2', res_block=4)
        x = self.residual_layer(x, out_dim=1024, layer_num='3', res_block=6)
        x = self.residual_layer(x, out_dim=2048, layer_num='4', res_block=3)

        x = Global_Average_Pooling(x)
        x = flatten(x)
        x = Linear(x)

        # x = tf.reshape(x, [-1,10])
        return x

# train_x, train_y, test_x, test_y = prepare_data()
# train_x, test_x = color_preprocessing(train_x, test_x)


# image_size = 32, img_channels = 3, class_num = 10 in cifar10
# os.environ['CUDA_VISIBLE_DEVICES'] = "1"
# os.environ['TF_CPP_MIN_LOG_LEVEL'] = "2"
# print(device_lib.list_local_devices())
#
# choose_device = "/device:GPU:0"
# with tf.device(choose_device):
#
#     x = tf.placeholder(tf.float32, shape=[None, image_depth, image_size, image_size, num_channels])
#     label = tf.placeholder(tf.float32, shape=[None, num_classes])
#
#     training_flag = tf.placeholder(tf.bool)
#
#     learning_rate = tf.placeholder(tf.float32, name='learning_rate')
#
#     with tf.name_scope("cross_entropy"):
#         logits = ResNeXt(x, training=training_flag).model
#         cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=label, logits=logits))
#         l2_loss = tf.add_n([tf.nn.l2_loss(var) for var in tf.trainable_variables()])
#
#     update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
#     with tf.control_dependencies(update_ops):
#         optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate, momentum=momentum, use_nesterov=True)
#         train = optimizer.minimize(cost + l2_loss * weight_decay)
#
#     with tf.name_scope("accuracy"):
#         correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(label, 1))
#         accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
#
#     # saver = tf.train.Saver(tf.global_variables())
#
#     gpu_options = tf.GPUOptions(allow_growth=True)
#     with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
#         # ckpt = tf.train.get_checkpoint_state('./model')
#         ckpt = 0
#         if ckpt and tf.train.checkpoint_exists(ckpt.model_checkpoint_path):
#             saver.restore(sess, ckpt.model_checkpoint_path)
#         else:
#             sess.run(tf.global_variables_initializer())
#
#         summary_writer = tf.summary.FileWriter('./../graphs', sess.graph)
#
#         epoch_learning_rate = init_learning_rate
#         print("Session started.")
#         for epoch in range(1, total_epochs + 1):
#             if epoch == (total_epochs * 0.5) or epoch == (total_epochs * 0.75):
#                 epoch_learning_rate = epoch_learning_rate / 10
#
#             pre_index = 0
#             train_acc = 0.0
#             train_loss = 0.0
#             iteration = 10
#             for step in range(1, iteration + 1):
#                 # if pre_index + batch_size < 50000:
#                 #     batch_x = train_x[pre_index: pre_index + batch_size]
#                 #     batch_y = train_y[pre_index: pre_index + batch_size]
#                 # else:
#                 #     batch_x = train_x[pre_index:]
#                 #     batch_y = train_y[pre_index:]
#                 print("{} iter started.".format(step))
#                 batch_x = np.ones((batch_size, image_depth, image_size, image_size, num_channels))
#                 batch_y = np.zeros((batch_size, num_classes))
#                 # batch_x = data_augmentation(batch_x)
#
#                 train_feed_dict = {
#                     x: batch_x,
#                     label: batch_y,
#                     learning_rate: epoch_learning_rate,
#                     training_flag: True
#                 }
#
#                 _, batch_loss = sess.run([train, cost], feed_dict=train_feed_dict)
#                 batch_acc = accuracy.eval(feed_dict=train_feed_dict)
#
#                 train_loss += batch_loss
#                 train_acc += batch_acc
#                 pre_index += batch_size
#
#                 print("{}-{}, epoch: {}, iter: {} done.".format(batch_loss, batch_acc, epoch, iteration))
#
#
#             train_loss /= iteration # average loss
#             train_acc /= iteration # average accuracy
#
#             train_summary = tf.Summary(value=[tf.Summary.Value(tag='train_loss', simple_value=train_loss),
#                                               tf.Summary.Value(tag='train_accuracy', simple_value=train_acc)])
#
#             # test_acc, test_loss, test_summary = Evaluate(sess)
#
#             summary_writer.add_summary(summary=train_summary, global_step=epoch)
#             # summary_writer.add_summary(summary=test_summary, global_step=epoch)
#             summary_writer.flush()
#
#             line = "epoch: %d/%d, train_loss: %.4f, train_acc: %.4f \n" % (epoch, total_epochs, train_loss, train_acc)
#             print(line)
#
#             # with open('logs.txt', 'a') as f:
#             #     f.write(line)
#
#             # saver.save(sess=sess, save_path='./model/ResNeXt.ckpt')
<file_sep>import sys

# insert at 1, 0 is the script path (or '' in REPL)
sys.path.insert(1, "/home/axp798/axp798gallinahome/Gesture-Detection/")
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import os
import pickle
from image_acquisition import getBoundingBox
import time

video_store = "/home/axp798/axp798gallinahome/jester_datase/20bn-jester-v1/"
json_store = "/home/axp798/axp798gallinahome/JD_JSON"
train_lables_path = '/home/axp798/axp798gallinahome/Gesture-Detection/jester-v1-train.csv'
labels_path = '/home/axp798/axp798gallinahome/Gesture-Detection/jester-v1-labels.csv'


def video_labels(train_lables_path, labels_path):
    '''

    :@param train_lables_path: path of file containing video names with their labels
    :@param labels_path: path of file containing index of each gesture

    :return: sorted list of video names along with their label (from 0 to 26)
    '''

    tpd = pd.read_csv(train_lables_path, sep=';', header=None)
    lab = pd.read_csv(labels_path, sep=';', header=None)

    arr_train = np.array(tpd)
    lst_train = list(arr_train)
    lst_train = [list(arr) for arr in lst_train]

    lst_train = sorted(lst_train, key=lambda x: x[0])

    lst_labels = np.array(lab)
    lst_labels = list(lst_labels)
    lst_labels = [list(arr)[0] for arr in lst_labels]

    ref_dict = {label: ind for ind, label in enumerate(lst_labels)}

    for ind, row in enumerate(lst_train):
        lst_train[ind][1] = ref_dict[row[1]]
        ind += 1

    return lst_train


def getFrameCut(frame, bounding_box):
    '''

    :param frame: numpy array of image
    :param bounding_box: list of bounds of the box
    :return: bounding box of the numpy array
    '''

    left_most = int(bounding_box[0])
    right_most = int(bounding_box[1])
    top_most = int(bounding_box[2])
    bottom_most = int(bounding_box[3])

    return frame[top_most: bottom_most, left_most: right_most]


def returnBoundingBox(json_path, img_height, img_width):
    '''

    :param json_path: path of folder containing json files of the current video
    :param img_height: height of image frame
    :param img_width: width of image frame
    :return: left, right top and bottom bounds
    '''

    global_val = [100000, 0, 100000, img_height]
    return getBoundingBox(json_path, global_val, img_height, img_width)


def getVideoDimension(video_path):
    '''

    :param video_name: name of the video for which dimension is asked
    :return: shape of the video frame
    '''

    frame_list = os.listdir(video_path)
    sample_image_path = os.path.join(video_path, frame_list[0])
    curr_image = plt.imread(sample_image_path)
    return curr_image.shape


def saveFramesWithLabels(video_store, json_store, train_lables_path, labels_path):
    '''

    :param video_store: path of folder containing all the videos
    :param json_store: path of the folder containing json files of all the videos
    :param train_lables_path: path of file containing video names with their labels
    :param labels_path: path of file containing index of each gesture
    :return: stores a dictionary with key as a list of numpy array of video frame and value as video label
    '''

    list_train_videos_name = video_labels(train_lables_path, labels_path)

    num_videos = 0

    net_start_time = time.time()
    log_file_path = "/home/axp798/axp798gallinahome/Gesture-Detection/assets/jes_dat/log1.txt"

    file1 = open(log_file_path, "w")
    file1.write("Logs: \n")
    file1.close()

    bounding_box_list = []
    incorrect_box = []

    save_video_path = '/home/axp798/axp798gallinahome/data/jester/train_64/'
    save_bounding_box_path = "/home/axp798/axp798gallinahome/data/jester/boxes/"

    for videoName_and_label in list_train_videos_name[start: end]:
        num_videos += 1
        video_start_time = time.time()
        video_name = videoName_and_label[0]
        label = videoName_and_label[1]

        video_name = str(video_name)

        modified_video_name = video_name.zfill(6)  # to get json files
        json_path = os.path.join(json_store, modified_video_name)
        video_path = os.path.join(video_store, video_name)

        print("{}-{}-{} started!".format(num_videos, video_name, modified_video_name))

        img_shape = getVideoDimension(video_path)
        img_height = img_shape[0]
        img_width = img_shape[1]

        bounding_box = returnBoundingBox(json_path, img_height, img_width)
        frame_list = os.listdir(os.path.join(video_store, video_name))

        cut_frame_list = []
        check = 0
        for ind in range(len(frame_list)):
            frame_name = str(ind + 1).zfill(5) + ".jpg"
            frame_path = os.path.join(video_store + video_name, frame_name)
            frame = plt.imread(frame_path)
            frame_cut = getFrameCut(frame, bounding_box)

            if frame_cut.size == 0:
                incorrect_box.append(modified_video_name)
                check = 1
                break

            res = cv2.resize(frame_cut, dsize=(64, 64), interpolation=cv2.INTER_CUBIC)
            cut_frame_list.append(res)

        if check:
            log1 = "{}-{}-{} ; incorrect boxes: {}-{}! \n".format(num_videos, video_name, modified_video_name,
                                                                  len(incorrect_box), incorrect_box)
            file1 = open(log_file_path, "a")
            file1.write(log1)
            file1.close()
            print(log1)
            continue

        cut_frame_array = np.array(cut_frame_list)
        video_shape = cut_frame_array.shape

        bounding_box_list.append(["{}".format(modified_video_name), bounding_box])

        pkl_vid_name = "{}_{}.pkl".format(num_videos, modified_video_name)

        with open(save_video_path + pkl_vid_name, 'wb') as f:
            pickle.dump([cut_frame_array, label], f)

        with open(save_bounding_box_path + "train_boxes.pkl", 'wb') as f:
            pickle.dump(bounding_box_list, f)

        video_end_time = time.time()

        log = "{}-{}-{} done in {}! ----- bounding box: {}, video shape: {} \n".format(num_videos, video_name,
                                                                                       modified_video_name,
                                                                                       video_end_time - video_start_time,
                                                                                       bounding_box, video_shape)
        file1 = open(log_file_path, "a")
        file1.write(log)
        file1.close()
        print(log)

    net_end_time = time.time()

    print("Done in {}s".format(net_end_time - net_start_time))


if __name__ == '__main__':
    saveFramesWithLabels(video_store, json_store, train_lables_path, labels_path)
<file_sep># -*- coding:utf-8 -*-

import os
import pickle
import random
import sys
import time

import cv2
import numpy as np
import scipy.ndimage


class MultiScaleCornerCrop(object):
    """Crop the given PIL.Image to randomly selected size.
    A crop of size is selected from scales of the original size.
    A position of cropping is randomly selected from 4 corners and 1 center.
    This crop is finally resized to given size.
    Args:
        scales: cropping scales of the original size
        size: size of the smaller edge
        interpolation: Default: PIL.Image.BILINEAR
    """

    def __init__(self,
                 scales,
                 shape,
                 interpolation=cv2.INTER_AREA,
                 crop_positions=('c', 'tl', 'tr', 'bl', 'br')):

        self.scales = scales
        self.shape = shape
        self.interpolation = interpolation
        self.scale = None
        self.crop_position = None
        self.crop_positions = crop_positions

    def __call__(self, img):
        min_length = min(img.shape[0], img.shape[1])
        crop_shape = int(min_length * self.scale)

        image_width = img.shape[0]
        image_height = img.shape[1]
        x1, x2, y1, y2 = 0, 0, 0, 0

        if self.crop_position == 'c':
            center_x = image_width // 2
            center_y = image_height // 2
            box_half = crop_shape // 2
            x1 = center_x - box_half
            y1 = center_y - box_half
            x2 = center_x + box_half
            y2 = center_y + box_half

        elif self.crop_position == 'tl':
            x1 = 0
            y1 = 0
            x2 = crop_shape
            y2 = crop_shape

        elif self.crop_position == 'tr':
            x1 = image_width - crop_shape
            y1 = 0
            x2 = image_width
            y2 = crop_shape

        elif self.crop_position == 'bl':
            x1 = 0
            y1 = image_height - crop_shape
            x2 = crop_shape
            y2 = image_height

        elif self.crop_position == 'br':
            x1 = image_width - crop_shape
            y1 = image_height - crop_shape
            x2 = image_width
            y2 = image_height

        img = img[y1: y2, x1: x2, :]

        return cv2.resize(img, dsize=(self.shape[0], self.shape[1]), interpolation=self.interpolation)

    def randomize_parameters(self):
        self.scale = self.scales[random.randint(0, len(self.scales) - 1)]
        self.crop_position = self.crop_positions[random.randint(0, len(self.scales) - 1)]


class SpatialElasticDisplacement(object):

    def __init__(self, sigma=2.0, alpha=1.0, order=0, cval=0, mode="constant"):
        self.alpha = alpha
        self.sigma = sigma
        self.order = order
        self.cval = cval
        self.mode = mode

    def __call__(self, img):
        if self.p < 0.50:
            image = img
            image_first_channel = image[:, :, 0]
            indices_x, indices_y = self._generate_indices(image_first_channel.shape, alpha=self.alpha, sigma=self.sigma)
            ret_image = (self._map_coordinates(
                image,
                indices_x,
                indices_y,
                order=self.order,
                cval=self.cval,
                mode=self.mode))

            return ret_image
        else:
            return img

    def _generate_indices(self, shape, alpha, sigma):
        assert (len(shape) == 2), "shape: Should be of size 2!"
        dx = scipy.ndimage.gaussian_filter((np.random.rand(*shape) * 2 - 1), sigma, mode="constant", cval=0) * alpha
        dy = scipy.ndimage.gaussian_filter((np.random.rand(*shape) * 2 - 1), sigma, mode="constant", cval=0) * alpha

        x, y = np.meshgrid(np.arange(shape[0]), np.arange(shape[1]), indexing='ij')
        return np.reshape(x + dx, (-1, 1)), np.reshape(y + dy, (-1, 1))

    def _map_coordinates(self, image, indices_x, indices_y, order=1, cval=0, mode="constant"):
        assert (len(image.shape) == 3), "image.shape: Should be of size 3!"
        result = np.copy(image)
        height, width = image.shape[0:2]
        for c in range(image.shape[2]):
            remapped_flat = scipy.ndimage.interpolation.map_coordinates(
                image[..., c],
                (indices_x, indices_y),
                order=order,
                cval=cval,
                mode=mode
            )
            remapped = remapped_flat.reshape((height, width))
            result[..., c] = remapped
        return result

    def randomize_parameters(self):
        self.p = random.random()
        self.alpha = 1 if random.random() > 0.5 else 2


def data_augmentation(batch, shape):
    mean = [119.65972197, 100.70736938, 107.86672802]
    std = [55.12904017, 52.20730189, 52.71543496]

    mscc = MultiScaleCornerCrop(scales=[1, 0.93, 0.87, 0.80], shape=shape)
    sed = SpatialElasticDisplacement()
    ans_batch = np.zeros((batch.shape[0], batch.shape[1], shape[0], shape[1], 3))
    for video_index in range(batch.shape[0]):
        curr_video = batch[video_index]
        mscc.randomize_parameters()
        sed.randomize_parameters()
        for frame_index in range(curr_video.shape[0]):
            curr_image = curr_video[frame_index]
            normalized_image = (curr_image - mean) / std
            normalized_image = normalized_image / 255.0
            cropped_and_scaled = mscc.__call__(normalized_image)
            ans_batch[video_index, frame_index, :, :, :] = sed.__call__(cropped_and_scaled)

    return ans_batch
<file_sep>import sys
sys.path.insert(1, "/home/axp798/axp798gallinahome/Gesture-Detection/")
import numpy as np
import matplotlib.pyplot as plt
import cv2
import os
import time

video_store = "/mnt/ConGD_phase_1/valid/"
labels_path = '/mnt/ConGD_labels/valid.txt'
store_frames_path = "/shared/valid/frames/"

with open(labels_path, 'r') as f:
    lines = f.readlines()

num_sample = 0
prev_mean = 0
prev_squared_mean = 0
prev_samples = -1
global_squared_mean = 0
global_mean = 0

for curr_line in lines:

    net_start_time = time.time()

    line_ele = curr_line.split(' ')
    dir_info = line_ele[0]
    par_name, vid_name = dir_info.split('/')[0], dir_info.split('/')[1]
    label_info = line_ele[1:]

    for j, curr_label in enumerate(label_info):

        video_start_time = time.time()
        curr_label_info = curr_label.split(':')

        rang = curr_label_info[0]
        label = curr_label_info[1]

        start_frame = rang.split(',')[0]
        end_frame = rang.split(',')[1]

        start_frame, end_frame = int(start_frame), int(end_frame)

        img_height = 240
        img_width = 320
        num_sample += 1
        print("{} video started.".format(num_sample))

        frame_list = []
        for ind in range(start_frame - 1, end_frame):
            frame_name = "frame{}.jpg".format(ind + 1)
            video_frame_dir = os.path.join(store_frames_path, "{}/{}".format(par_name, vid_name))
            frame_path = os.path.join(video_frame_dir, frame_name)
            frame = plt.imread(frame_path)
            frame_list.append(frame)

        frame_array = np.array(frame_list)
        curr_mean = np.mean(frame_array, axis=(0, 1, 2))
        curr_samples = len(frame_list) * img_height * img_width
        if prev_samples != -1:
            multiply_factor = (curr_samples / (curr_samples + prev_samples), prev_samples / (curr_samples + prev_samples))
            global_mean = curr_mean * multiply_factor[0] + prev_mean * multiply_factor[1]
            prev_samples += curr_samples
        else:
            global_mean = curr_mean
            prev_samples = curr_samples

        prev_mean = global_mean
        print(prev_mean)

global_channel_mean = global_mean

num_sample = 0
for curr_line in lines:

    net_start_time = time.time()

    line_ele = curr_line.split(' ')
    dir_info = line_ele[0]
    par_name, vid_name = dir_info.split('/')[0], dir_info.split('/')[1]
    label_info = line_ele[1:]

    for j, curr_label in enumerate(label_info):
        # TODO:
        # get bounding box
        # convert to frames
        # get cut frames

        video_start_time = time.time()
        curr_label_info = curr_label.split(':')

        rang = curr_label_info[0]
        label = curr_label_info[1]

        start_frame = rang.split(',')[0]
        end_frame = rang.split(',')[1]

        start_frame, end_frame = int(start_frame), int(end_frame)

        img_height = 240
        img_width = 320
        num_sample += 1
        print("{} video started.".format(num_sample))

        frame_list = []
        for ind in range(start_frame - 1, end_frame):
            frame_name = "frame{}.jpg".format(ind + 1)
            video_frame_dir = os.path.join(store_frames_path, "{}/{}".format(par_name, vid_name))
            frame_path = os.path.join(video_frame_dir, frame_name)
            frame = plt.imread(frame_path)
            frame_list.append(frame - global_channel_mean)

        frame_array = np.array(frame_list)
        curr_squared_mean = np.mean(np.square(frame_array), axis=(0, 1, 2))
        curr_samples = len(frame_list) * img_height * img_width
        if prev_samples != -1:
            multiply_factor = (curr_samples / (curr_samples + prev_samples), prev_samples / (curr_samples + prev_samples))
            global_squared_mean = curr_squared_mean * multiply_factor[0] + prev_squared_mean * multiply_factor[1]
            prev_samples += curr_samples
        else:
            global_squared_mean = curr_squared_mean
            prev_samples = curr_samples

        prev_squared_mean = global_squared_mean
        print(prev_squared_mean)

global_channel_std = np.sqrt(prev_squared_mean)
print(global_channel_mean)
print(global_channel_std)

<file_sep>import sys

sys.path.insert(1, "/home/axp798/axp798gallinahome/Gesture-Detection/")
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import os
import pickle
import image_acquisition2 as img_ac
import time

video_store = "/home/axp798/axp798gallinahome/ConGD/ConGD_phase_1/train/"
json_store = "/home/axp798/axp798gallinahome/store/train/json/"
labels_path = '/home/axp798/axp798gallinahome/ConGD/ConGD_labels/train.txt'
store_frames_path = "/home/axp798/axp798gallinahome/store/train/frames/"


def store_frames(dir_name, vid_name):
    vid_path = os.path.join(video_store, "{}/{}.M.avi".format(dir_name, vid_name))

    video_par_dir = os.path.join(store_frames_path, dir_name)
    video_frame_dir = os.path.join(store_frames_path, "{}/{}".format(dir_name, vid_name))

    if not os.path.exists(video_par_dir):
        os.mkdir(video_par_dir)

    if not os.path.exists(video_frame_dir):
        os.mkdir(video_frame_dir)

    vidcap = cv2.VideoCapture(vid_path)
    success, image = vidcap.read()
    count = 1
    while success:
        cv2.imwrite(os.path.join(video_frame_dir, "frame{}.jpg".format(count)), image)  # save frame as JPEG file
        success, image = vidcap.read()
        count += 1


def getFrameCut(frame, bounding_box):
    '''

    :param frame: numpy array of image
    :param bounding_box: list of bounds of the box
    :return: bounding box of the numpy array
    '''

    left_most = int(bounding_box[0])
    right_most = int(bounding_box[1])
    top_most = int(bounding_box[2])
    bottom_most = int(bounding_box[3])

    return frame[top_most: bottom_most, left_most: right_most]


def returnBoundingBox(json_path, img_height, img_width):
    '''

    :param json_path: path of folder containing json files of the current video
    :param img_height: height of image frame
    :param img_width: width of image frame
    :return: left, right top and bottom bounds
    '''

    global_val = [100000, 0, 100000, img_height]
    return getBoundingBox(json_path, global_val, img_height, img_width)


def getVideoDimension(video_path):
    '''

    :param video_name: name of the video for which dimension is asked
    :return: shape of the video frame
    '''

    frame_list = os.listdir(video_path)
    sample_image_path = os.path.join(video_path, frame_list[0])
    curr_image = plt.imread(sample_image_path)
    return curr_image.shape


def saveFramesWithLabels(video_store, json_store, labels_path):
    '''

    :param video_store: path of folder containing all the videos
    :param json_store: path of the folder containing json files of all the videos
    :param train_lables_path: path of file containing video names with their labels
    :param labels_path: path of file containing index of each gesture
    :return: stores a dictionary with key as a list of numpy array of video frame and value as video label
    '''

    num_videos = 0

    start_time = time.time()

    save_video_path = '/home/axp798/axp798gallinahome/data/chalearn/train_112/'
    save_bounding_box_path = "/home/axp798/axp798gallinahome/data/chalearn/train_boxes/"
    log_file_path = "/home/axp798/axp798gallinahome/Gesture-Detection/assets/chalearn_dataset/log1.txt"

    file1 = open(log_file_path, "w")
    file1.write("Logs: \n")
    file1.close()

    bounding_box_list = []
    incorrect_box = []

    with open(labels_path, 'r') as f:
        lines = f.readlines()

    for curr_line in lines:

        net_start_time = time.time()

        line_ele = curr_line.split(' ')
        dir_info = line_ele[0]
        par_name, vid_name = dir_info.split('/')[0], dir_info.split('/')[1]
        label_info = line_ele[1:]

        for j, curr_label in enumerate(label_info):
            # TODO:
            # get bounding box
            # convert to frames
            # get cut frames

            num_videos += 1
            video_start_time = time.time()
            curr_label_info = curr_label.split(':')

            rang = curr_label_info[0]
            label = curr_label_info[1]

            start_frame = rang.split(',')[0]
            end_frame = rang.split(',')[1]

            start_frame, end_frame = int(start_frame), int(end_frame)

            # Get bounding box
            dir_path = os.path.join(json_store, par_name)
            vid_path = os.path.join(dir_path, vid_name)

            img_height = 240
            img_width = 320

            json_path = vid_path
            global_val = [100000, 0, 100000, img_height]
            bounding_box = img_ac.getBoundingBox(json_path, vid_name, global_val, img_height, img_width, start_frame,
                                                 end_frame)

            # store frames
            if j == 0:
                store_frames(par_name, vid_name)

            # get cut frames
            cut_frame_list = []
            check = 0
            for ind in range(start_frame - 1, end_frame):
                frame_name = "frame{}.jpg".format(ind + 1)
                video_frame_dir = os.path.join(store_frames_path, "{}/{}".format(par_name, vid_name))
                frame_path = os.path.join(video_frame_dir, frame_name)
                frame = plt.imread(frame_path)
                frame_cut = getFrameCut(frame, bounding_box)

                if frame_cut.size == 0:
                    incorrect_box.append([dir_info, start_frame, end_frame])
                    check = 1
                    break

                res = cv2.resize(frame_cut, dsize=(112, 112), interpolation=cv2.INTER_CUBIC)
                cut_frame_list.append(res)

            if check:
                log1 = "{}-{} ; incorrect boxes: {}-{}! \n".format(num_videos, dir_info,
                                                                   len(incorrect_box), incorrect_box)
                file1 = open(log_file_path, "a")
                file1.write(log1)
                file1.close()
                print(log1)
                continue

            cut_frame_array = np.array(cut_frame_list)
            video_shape = cut_frame_array.shape

            bounding_box_list.append(["{}_{}-{}".format(dir_info, start_frame, end_frame), bounding_box, label])

            pkl_vid_name = "{}-{}-{}_{}-{}.pkl".format(num_videos, par_name, vid_name, start_frame, end_frame)

            with open(save_video_path + pkl_vid_name, 'wb') as f:
                pickle.dump([cut_frame_array, label], f)

            with open(save_bounding_box_path + "boxes.pkl", 'wb') as f:
                pickle.dump(bounding_box_list, f)

            video_end_time = time.time()

            log = "{}-{}-{} done in {}s ----- bounding box: {}, video shape: {} \n".format(num_videos, dir_info,
                                                                                           "{}_{}".format(start_frame,
                                                                                                          end_frame),
                                                                                           video_end_time - video_start_time,
                                                                                           bounding_box, video_shape)
            file1 = open(log_file_path, "a")
            file1.write(log)
            file1.close()
            print(log)

        net_end_time = time.time()

        print("########### Line {} done in {}s ###############".format(dir_info, net_end_time - net_start_time))

    end_time = time.time()
    print("Done in {}s".format(end_time - start_time))


if __name__ == '__main__':
    saveFramesWithLabels(video_store, json_store, labels_path)
<file_sep># Importing Libraries
import os
import cv2

# Path to openpose singularity image being used
openpose_simg = "/home/axp798/axp798gallinahome/openpose_1_0_5.simg"  

# Path to directory for storing rendered videos and json files from train videos
store_path = "/home/axp798/axp798gallinahome/store/train/"  

# directory to store rendered videos in the same format as train videos are stored
videos_store = os.path.join(store_path, "videos/")

# directory to store json files containing hand, face and body coordinates from the videos
json_store = os.path.join(store_path, "json/")  

# Path to list of directories containing training videos
train_dir = "/home/axp798/axp798gallinahome/ConGD/ConGD_phase_1/train/"  
video_bind = ""
lib_dir = "/home/axp798/axp798gallinahome/"  # Path to directory to be accessible from within the singularity container


def create_frames(video_path, dir_name, video_name, parts):

    # Creates and stores video frames from the given video

    vidcap = cv2.VideoCapture(video_path)
    done, frame = vidcap.read()
    num = 0
    if not os.path.exists(os.path.join(store_path, "frames/{}/{}/".format(dir_name, parts[0]))):
        # create directory to store frames
        os.mkdir(os.path.join(store_path, "frames/{}/{}/".format(dir_name, parts[0])))
    while done:
        frame_store = os.path.join(store_path, "frames/{}/{}/frames{}.jpg".format(dir_name, parts[0], num))
        cv2.imwrite(frame_store, frame)     # save frame as JPEG file
        done, frame = vidcap.read()
        print('Read a new frame: ', done)

        num += 1

    return

def delete_video(write_video_path):
    os.system("rm {}".format(write_video_path))  # execute the delete video command
    return

def main():
    # Uses the videos in the train_dir to extract json files from them using openpose, 
	# stores videos having coordinates in them, creates frames from them and deletes the videos finally

    dir_ptr = 15

    for dir_name in os.listdir(train_dir)[14: 100]:

        print("Directory {} started!".format(dir_ptr))
        # dir_name = 001, 002, ..... , 027, 028, ..... , 128, 129, ....
        print("{} directory started ".format(dir_name))

        # video_dir = "/home/axp798/axp798gallinahome/ConGD/ConGD_phase_1/train/001/"
        video_dir = os.path.join(train_dir, dir_name)

        if not os.path.exists(os.path.join(store_path, "frames/{}/".format(dir_name))):
            os.mkdir(os.path.join(store_path, "frames/{}/".format(dir_name)))

        if not os.path.exists(os.path.join(videos_store, "{}/".format(dir_name))):
            # creating the directory to store videos
            # os.path.join(videos_store, "{}/".format(dir_name)) = "/home/axp798/axp798gallinahome/store/train/videos/001/"

            os.mkdir(os.path.join(videos_store, "{}/".format(dir_name)))

        if not os.path.exists(os.path.join(json_store, "{}/".format(dir_name))):
            os.mkdir(os.path.join(json_store, "{}/".format(dir_name)))

        video_ptr = 1

        for video_name in os.listdir(video_dir):
            # video_name = 00001.K.avi, 00001.M.avi etc.
            # .k extension is for depth frames videos
            # .M extension videos is for 3D RGB videos

            parts = video_name.split('.')  # parts = [00001, k, avi] or parts = [00001, M, avi]

            if (parts[1] == 'K'):
                # excluding the depth frames videos
                continue

            else:
                # prcessing the RGB videos
                print("Directory {} and video {} started!".format(dir_ptr, video_ptr))
                print("{} video directory started ".format(video_name))

                add = "{}/{}".format(dir_name, video_name)  # add = "001/00001.M.avi"
                video_path = os.path.join(train_dir, add)
                # video_path = "/home/axp798/axp798gallinahome/ConGD/ConGD_phase_1/train/001/00001.M.avi"
                print(video_path)

                # write_video_path = "/home/axp798/axp798gallinahome/store/train/videos/001/result_00001.avi"
                write_video_path = os.path.join(videos_store, "{}/result_{}.avi".format(dir_name, parts[0]))

                # write_json_path = "/home/axp798/axp798gallinahome/store/train/json/001/00001"
                write_json_path = os.path.join(json_store, "{}/{}/".format(dir_name, parts[0]))

                video_bind = video_path
                
                print("command started!")
                command = "singularity run --nv --bind {},{} {} --video {} --write_keypoint_json {} --no_display " \
                          "--render_pose 0 " \
                          "--hand"\
                    .format(
                    video_path,
                    lib_dir,
                    openpose_simg,
                    video_path,
                    write_json_path)

                os.system(command)  # run the command on terminal
				
                # convert the rendered videos with openpose coordinates to frames
                create_frames(write_video_path, dir_name, video_name, parts)   

                # delete the rendered videos to save disk space
				delete_video(write_video_path)  
					
                video_ptr += 1
                print("Directory {} and video {} completed!".format(dir_ptr, video_ptr))

        dir_ptr += 1
        print("Directory {} completed!".format(dir_ptr))


if __name__ == '__main__':
    main()
<file_sep>import argparse
import os
import pickle
import random
import sys
import time

from resnext import ResNeXt
import c3D as conv3d4
import c3D2 as conv3d2
import c3D3 as conv3d3
import c3D_main as conv3d1
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from chalearn_utils import data_augmentation
from tensorflow.python.client import device_lib


# from AdamWOptimizer import create_optimizer
# from tensorflow.keras import *
# from tensorflow.keras.layers import *
# from tensorflow.keras.models import *


def append_frames(cut_frame_array, required_num_frames):
    appended_list_of_frames = list(cut_frame_array)

    num_curr_frames = cut_frame_array.shape[0]
    num_more_frames = required_num_frames - num_curr_frames

    for i in range(num_more_frames):
        appended_list_of_frames.append(cut_frame_array[i % num_curr_frames])

    return np.array(appended_list_of_frames)


def get_video_reduced_fps(appended_video, factor):
    num_total_frames = appended_video.shape[0]
    final_video = []
    for index in range(0, num_total_frames, factor):
        final_video.append(appended_video[index])

    return np.array(final_video)


def get_final_video(par_name, vid_name, label_info, num_video_in_curr_line, window_size, set='train'):

    curr_label = label_info[num_video_in_curr_line]
    curr_label_info = curr_label.split(':')

    rang = curr_label_info[0]
    label = curr_label_info[1]

    start_frame = rang.split(',')[0]
    end_frame = rang.split(',')[1]

    start_frame, end_frame = int(start_frame), int(end_frame)

    frame_list = []

    for ind in range(start_frame - 1, end_frame):
        frame_name = "frame{}.jpg".format(ind + 1)
        if set == 'train':
            video_frame_dir = os.path.join(train_frames_path, "{}/{}".format(par_name, vid_name))
        else:
            video_frame_dir = os.path.join(valid_frames_path, "{}/{}".format(par_name, vid_name))
        frame_path = os.path.join(video_frame_dir, frame_name)
        frame = plt.imread(frame_path)
        frame_list.append(frame)

    frame_array = np.array(frame_list)
    num_frames = frame_array.shape[0]

    required_num_frames = (int(num_frames / window_size) + 1) * window_size
    appended_video = append_frames(frame_array, required_num_frames)  # returns numpy array
    factor = int(appended_video.shape[0] / window_size)
    final_video = get_video_reduced_fps(appended_video, factor)  # returns numpy array

    return final_video, label


def extract_line_info(curr_line):
    line_ele = curr_line.split(' ')
    dir_info = line_ele[0]
    par_name, vid_name = dir_info.split('/')[0], dir_info.split('/')[1]
    label_info = line_ele[1:]
    # total_video_in_curr_line = len(label_info)

    return par_name, vid_name, label_info


def train_neural_network(x_inpuT,
                         y_inpuT,
                         labels_path,
                         val_labels_path,
                         save_loss_path,
                         save_model_path,
                         batch_size,
                         val_batch_size,
                         image_height,
                         image_width,
                         learning_rate,
                         weight_decay,
                         num_iter,
                         epochs,
                         which_model,
                         num_train_videos,
                         num_val_videos,
                         win_size):

    with tf.name_scope("cross_entropy"):

        prediction = 0
        shapes_list = None
        if which_model == 1:
            prediction, shapes_list = conv3d1.inference(x_inpuT)

        elif which_model == 2:
            resnext_model = ResNeXt(x_inpuT, tf.constant(True, dtype=tf.bool))
            prediction = resnext_model.Build_ResNext(x_inpuT)
        elif which_model == 3:
            prediction = conv3d2.inference(x_inpuT)
        elif which_model == 4:
            prediction = conv3d3.inference(x_inpuT)
        elif which_model == 5:
            prediction, shapes_list = conv3d4.inference(x_inpuT)

        cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=y_inpuT))

    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):

        # optimizer = 0
        if weight_decay is not None:
            print("weight decay applied.")
            optimizer = create_optimizer(cost, learning_rate, weight_decay)
        else:
            optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)

    with tf.name_scope("accuracy"):
        correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y_inpuT, 1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))

    # saver = tf.train.Saver(save_relative_paths=True)
    print("Calculating total parameters in the model")
    total_parameters = 0
    for variable in tf.trainable_variables():
        # shape is an array of tf.Dimension
        shape = variable.get_shape()
        print(shape)
        # print(len(shape))
        variable_parameters = 1
        for dim in shape:
            # print(dim)
            variable_parameters *= dim.value
        # print(variable_parameters)
        total_parameters += variable_parameters
    print(total_parameters)
    if shapes_list is not None:
        print(shapes_list)

    gpu_options = tf.GPUOptions(allow_growth=True)
    with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:

        print("session starts!")

        sess.run(tf.global_variables_initializer())

        start_time = time.time()
        epoch_loss_list = []
        val_loss_list = []

        ori_height = 240
        ori_width = 320

        with open(labels_path, 'r') as f:
            lines = f.readlines()

        with open(val_labels_path, 'r') as f:
            val_lines = f.readlines()

        for epoch in range(epochs):

            print("Epoch {} started!".format(epoch + 1))
            epoch_start_time = time.time()

            epoch_loss = 0
            train_acc = 0

            window_size = win_size

            random.seed(7)
            print("Random seed fixed for training.")

            num_videos = num_train_videos
            num_batches = int(num_videos / batch_size)
            num_iter_total_data_per_epoch = num_iter

            for iter_index in range(num_iter_total_data_per_epoch):

                prev_line = -1
                num_line = 0
                num_video_in_curr_line = 0

                # for batch_num in range(num_batches):

                batch_x = np.zeros((batch_size, depth, ori_height, ori_width, 3))
                batch_y = np.zeros((batch_size, num_classes))
                batch_index = 0
                num_batch_completed = 0

                while True:
                    if prev_line != num_line:
                        curr_line = lines[num_line]
                        par_name, vid_name, label_info = extract_line_info(curr_line)
                        total_video_in_curr_line = len(label_info)

                    final_video, label = get_final_video(par_name, vid_name, label_info, num_video_in_curr_line, window_size, set='train')
                    # print(final_video.shape)

                    batch_x[batch_index, :, :, :, :] = final_video

                    basic_line = [0] * num_classes
                    basic_line[int(label) - 1] = 1
                    basic_label = basic_line
                    batch_y[batch_index, :] = np.array(basic_label)
                    batch_index += 1

                    if batch_index == batch_size:
                        # train_batch
                        batch_start_time = time.time()

                        mini_batch_x = data_augmentation(batch_x, (image_height, image_width))
                        # mini_batch_x = mini_batch_x / 255.0
                        mini_batch_y = batch_y

                        perm = np.random.permutation(batch_size)
                        mini_batch_x = mini_batch_x[perm]
                        mini_batch_y = mini_batch_y[perm]

                        _optimizer, _cost, _prediction, _accuracy = sess.run([optimizer, cost, prediction, accuracy],
                                                                             feed_dict={x_inpuT: mini_batch_x,
                                                                                        y_inpuT: mini_batch_y})
                        epoch_loss += _cost
                        train_acc += _accuracy

                        num_batch_completed += 1
                        batch_end_time = time.time()

                        total_train_batch_completed = iter_index * num_batches + num_batch_completed

                        log1 = "\rEpoch: {}, " \
                               "iter: {}, " \
                               "batches completed: {}, " \
                               "time taken: {:.5f}, " \
                               "loss: {:.6f}, " \
                               "accuracy: {:.4f} \n". \
                            format(
                            epoch + 1,
                            iter_index + 1,
                            total_train_batch_completed,
                            batch_end_time - batch_start_time,
                            epoch_loss / (batch_size * total_train_batch_completed),
                            _accuracy)

                        print(log1)
                        sys.stdout.flush()

                        if num_batch_completed == num_batches:
                            break

                        batch_index = 0
                        batch_x = np.zeros((batch_size, depth, ori_height, ori_width, 3))
                        batch_y = np.zeros((batch_size, num_classes))

                    prev_line = num_line
                    num_video_in_curr_line += 1
                    if num_video_in_curr_line == total_video_in_curr_line:
                        num_video_in_curr_line = 0
                        num_line += 1

            # validation loss
            print("<---------------- Validation Set started ---------------->")
            val_loss = 0
            val_acc = 0

            val_num_videos = num_val_videos
            val_num_batches = int(val_num_videos / val_batch_size)

            random.seed(23)
            print("Random seed fixed for validation.")

            for __ in range(num_iter_total_data_per_epoch):
                prev_line = -1
                num_line = 0
                num_video_in_curr_line = 0

                # for batch_num in range(val_num_batches):
                #
                val_batch_x = np.zeros((val_batch_size, depth, ori_height, ori_width, 3))
                val_batch_y = np.zeros((val_batch_size, num_classes))
                batch_index = 0
                val_num_batch_completed = 0

                while True:
                    if prev_line != num_line:
                        curr_line = val_lines[num_line]
                        par_name, vid_name, label_info = extract_line_info(curr_line)
                        total_video_in_curr_line = len(label_info)

                    # process video
                    final_video, label = get_final_video(par_name, vid_name, label_info, num_video_in_curr_line, window_size, set='valid')
                    # print(final_video.shape)

                    val_batch_x[batch_index, :, :, :, :] = final_video

                    basic_line = [0] * num_classes
                    basic_line[int(label) - 1] = 1
                    basic_label = basic_line
                    val_batch_y[batch_index, :] = np.array(basic_label)
                    batch_index += 1

                    if batch_index == val_batch_size:
                        val_batch_x = data_augmentation(val_batch_x, (image_height, image_width))
                        # val_batch_x = val_batch_x / 255.0

                        perm = np.random.permutation(batch_size)
                        val_batch_x = val_batch_x[perm]
                        val_batch_y = val_batch_y[perm]

                        val_cost, val_batch_accuracy = sess.run([cost, accuracy],
                                                                feed_dict={x_inpuT: val_batch_x, y_inpuT: val_batch_y})

                        val_acc += val_batch_accuracy
                        val_loss += val_cost

                        val_num_batch_completed += 1

                        if val_num_batch_completed == val_num_batches:
                            break

                        val_batch_x = np.zeros((val_batch_size, depth, ori_height, ori_width, 3))
                        val_batch_y = np.zeros((val_batch_size, num_classes))
                        batch_index = 0

                    prev_line = num_line
                    num_video_in_curr_line += 1
                    if num_video_in_curr_line == total_video_in_curr_line:
                        num_video_in_curr_line = 0
                        num_line += 1

            epoch_end_time = time.time()

            total_train_batch_completed = num_iter_total_data_per_epoch * num_batch_completed
            total_val_num_batch_completed = num_iter_total_data_per_epoch * val_num_batch_completed

            epoch_loss = epoch_loss / (batch_size * total_train_batch_completed)
            train_acc = train_acc / total_train_batch_completed

            val_loss /= (val_batch_size * total_val_num_batch_completed)
            val_acc = val_acc / total_val_num_batch_completed

            log3 = "Epoch {} done; " \
                   "Time Taken: {:.4f}s; " \
                   "Train_loss: {:.6f}; " \
                   "Val_loss: {:.6f}; " \
                   "Train_acc: {:.4f}; " \
                   "Val_acc: {:.4f}; " \
                   "Train batches: {}; " \
                   "Val batches: {};\n". \
                format(epoch + 1, epoch_end_time - epoch_start_time, epoch_loss, val_loss, train_acc, val_acc,
                       num_iter_total_data_per_epoch * num_batch_completed,
                       num_iter_total_data_per_epoch * val_num_batch_completed)

            print(log3)

            if save_loss_path is not None:
                file1 = open(save_loss_path, "a")
                file1.write(log3)
                file1.close()

            epoch_loss_list.append(epoch_loss)
            val_loss_list.append(val_loss)

            if save_model_path is not None:
                saver.save(sess, save_model_path)

        end_time = time.time()
        print('Time elapse: ', str(end_time - start_time))
        print(epoch_loss_list)

        if save_loss_path is not None:
            file1 = open(save_loss_path, "a")
            file1.write("Train Loss List: {} \n".format(str(epoch_loss_list)))
            file1.write("Val Loss List: {} \n".format(str(val_loss_list)))
            file1.close()


if __name__ == '__main__':

    np.random.seed(0)
    parser = argparse.ArgumentParser()

    parser.add_argument('-cs', action='store', dest='check_singularity', type=int)
    parser.add_argument('-ih', action='store', dest='height', type=int)
    parser.add_argument('-iw', action='store', dest='width', type=int)
    parser.add_argument('-bs', action='store', dest='batch_size', type=int)
    parser.add_argument('-vbs', action='store', dest='val_batch_size', type=int)
    parser.add_argument('-lr', action='store', dest='learning_rate', type=float)
    parser.add_argument('-wd', action='store', dest='weight_decay', type=float, const=None)
    parser.add_argument('-ni', action='store', dest='num_iter', type=int)
    parser.add_argument('-e', action='store', dest='epochs', type=int)
    parser.add_argument('-ntv', action='store', dest='num_train_videos', type=int)
    parser.add_argument('-nvv', action='store', dest='num_val_videos', type=int)
    parser.add_argument('-ws', action='store', dest='win_size', type=int)
    parser.add_argument('-slp', action='store', dest='save_loss_path', const=None)
    parser.add_argument('-smp', action='store', dest='save_model_path', const=None)
    parser.add_argument('-mn', action='store', dest='model_num', type=int)
    parser.add_argument('-vd', action='store', dest='visible_devices')
    parser.add_argument('-nd', action='store', dest='num_device', type=int)

    results = parser.parse_args()

    arg_check_singularity = results.check_singularity
    arg_height = results.height
    arg_width = results.width
    arg_batch_size = results.batch_size
    arg_val_batch_size = results.val_batch_size
    arg_lr = results.learning_rate
    arg_wd = results.weight_decay
    arg_num_iter = results.num_iter
    arg_epochs = results.epochs
    arg_num_val_videos = results.num_val_videos
    arg_num_train_videos = results.num_train_videos
    arg_win_size = results.win_size
    arg_save_loss_path = results.save_loss_path
    arg_save_model_path = results.save_model_path
    arg_model_num = results.model_num
    arg_visible_devices = results.visible_devices
    arg_num_device = results.num_device

    labels_path = '/home/axp798/axp798gallinahome/ConGD/ConGD_labels/train.txt'
    val_labels_path = '/home/axp798/axp798gallinahome/ConGD/ConGD_labels/valid.txt'
    train_frames_path = "/home/axp798/axp798gallinahome/store/train/frames/"
    valid_frames_path = "/home/axp798/axp798gallinahome/store/valid/frames/"
    save_loss = "/home/axp798/axp798gallinahome/Gesture-Detection/models/loss_chalearn/"
    save_model = "/home/axp798/axp798gallinahome/Gesture-Detection/models/saved_models_chalearn/"

    if arg_check_singularity:
        labels_path = '/mnt/ConGD_labels/train.txt'
        val_labels_path = '/mnt/ConGD_labels/valid.txt'
        train_frames_path = "/shared/train/frames/"
        valid_frames_path = "/shared/valid/frames/"
        save_loss = "/home/models/loss_chalearn/"
        save_model = "/home/models/saved_models_chalearn/"

    ar_save_loss_path = None
    if arg_save_loss_path is not None:
        ar_save_loss_path = save_loss + "{}".format(arg_save_loss_path)

    ar_save_model_path = None
    if arg_save_model_path is not None:
        path = save_model + "{}/".format(arg_save_model_path)
        if not os.path.exists(path):
            os.mkdir(path)

        ar_save_model_path = path + "model"

    if ar_save_loss_path is not None:
        file1 = open(ar_save_loss_path, "w")
        file1.write("Params: {} \n".format(results))
        file1.write("Losses: \n")
        file1.close()

    depth = arg_win_size
    num_classes = 249

    os.environ['CUDA_VISIBLE_DEVICES'] = "{}".format(arg_visible_devices)
    os.environ['TF_CPP_MIN_LOG_LEVEL'] = "2"
    print(device_lib.list_local_devices())

    choose_device = "/device:GPU:{}".format(arg_num_device)
    tf.reset_default_graph()
    with tf.device(choose_device):
        x_inpuT = tf.placeholder(tf.float32, shape=[arg_batch_size, depth, arg_height, arg_width, 3])
        y_inpuT = tf.placeholder(tf.float32, shape=[arg_batch_size, num_classes])

        train_neural_network(x_inpuT, y_inpuT, labels_path, val_labels_path,
                             save_loss_path=ar_save_loss_path,
                             save_model_path=ar_save_model_path,
                             batch_size=arg_batch_size,
                             val_batch_size=arg_val_batch_size,
                             image_height=arg_height,
                             image_width=arg_width,
                             learning_rate=arg_lr,
                             weight_decay=arg_wd,
                             num_iter=arg_num_iter,
                             epochs=arg_epochs,
                             which_model=arg_model_num,
                             num_train_videos=arg_num_train_videos,
                             num_val_videos=arg_num_val_videos,
                             win_size=arg_win_size)

<file_sep>import os

import cv2
import matplotlib.pyplot as plt
import numpy as np


def getFrameCut(frame, bounding_box):
    '''

    :param frame: numpy array of image
    :param bounding_box: list of bounds of the box
    :return: bounding box of the numpy array
    '''

    left_most = int(bounding_box[0])
    right_most = int(bounding_box[1])
    top_most = int(bounding_box[2])
    bottom_most = int(bounding_box[3])

    return frame[top_most: bottom_most, left_most: right_most]


def get_frame_array(lst_item, video_store, image_height, image_width):
    '''

    :param lst_item: [video_name, bounding_box, label]
    :param video_store: path to jester videos
    :param image_height: required height of each frame
    :param image_width: required width of each frame
    :return: frame array with given bounding box and image dimensions
    '''

    mod_video_name = lst_item[0]
    video_name = mod_video_name.lstrip("0")
    bounding_box = lst_item[1]
    frame_list = os.listdir(os.path.join(video_store, video_name))

    cut_frame_list = []
    for ind in range(len(frame_list)):
        frame_name = str(ind + 1).zfill(5) + ".jpg"
        frame_path = os.path.join(video_store + video_name, frame_name)
        frame = plt.imread(frame_path)
        frame_cut = getFrameCut(frame, bounding_box)

        res = cv2.resize(frame_cut, dsize=(image_height, image_width), interpolation=cv2.INTER_CUBIC)
        cut_frame_list.append(res)

    cut_frame_array = np.array(cut_frame_list)

    return cut_frame_array


<file_sep>import argparse
import os
import pickle
import random
import sys
import time

import numpy as np
import resnet_10 as resnet
import tensorflow as tf
from AdamWOptimizer import create_optimizer
from tensorflow.python.client import device_lib


def append_frames(cut_frame_array, required_num_frames):
    appended_list_of_frames = list(cut_frame_array)

    num_curr_frames = cut_frame_array.shape[0]
    num_more_frames = required_num_frames - num_curr_frames

    for i in range(num_more_frames):
        appended_list_of_frames.append(cut_frame_array[i % num_curr_frames])

    return np.array(appended_list_of_frames)


def train_neural_network(x_inpuT,
                         y_inpuT,
                         data_path,
                         val_data_path,
                         save_loss_path,
                         save_model_path,
                         batch_size,
                         val_batch_size,
                         learning_rate,
                         weight_decay,
                         epochs,
                         which_model,
                         num_val_videos,
                         random_clips,
                         win_size,
                         ignore_factor):
    with tf.name_scope("cross_entropy"):

        prediction = 0
        if which_model == 1:
            prediction = resnet.inference(x_inpuT)

        cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=y_inpuT))

    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):

        # optimizer = 0
        if weight_decay is not None:
            print("weight decay applied.")
            optimizer = create_optimizer(cost, learning_rate, weight_decay)
        else:
            optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)

    with tf.name_scope("accuracy"):
        correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y_inpuT, 1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))

    saver = tf.train.Saver()

    gpu_options = tf.GPUOptions(allow_growth=True)
    with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:

        print("session starts!")

        sess.run(tf.global_variables_initializer())

        start_time = time.time()
        epoch_loss_list = []
        val_loss_list = []

        pkl_files_list = os.listdir(data_path)
        val_pkl_files_list = os.listdir(val_data_path)

        for epoch in range(epochs):

            print("Epoch {} started!".format(epoch + 1))
            epoch_start_time = time.time()

            epoch_loss = 0
            train_acc = 0
            num_batch_completed = 0

            window_size = win_size
            num_clips_per_video = random_clips

            random.seed(7)
            print("Random seed fixed for training.")

            # pkl_files_list = pkl_files_list[ : 64]
            num_videos = len(pkl_files_list) - (len(pkl_files_list) % batch_size)
            num_blocks = int(num_videos / batch_size)

            block_x = np.zeros((num_clips_per_video, batch_size, depth, height, width, 3))
            block_y = np.zeros((num_clips_per_video, batch_size, num_classes))

            for block_index in range(num_blocks):

                for index_in_batch, pkl_file in enumerate(
                        pkl_files_list[block_index * batch_size: (block_index + 1) * batch_size]):

                    with open(os.path.join(data_path, pkl_file), 'rb') as f:
                        frames_and_label = pickle.load(f)

                    cut_frame_array = frames_and_label[0]
                    label = frames_and_label[1]

                    num_frames = cut_frame_array.shape[0]
                    required_num_frames = int(window_size * ignore_factor)

                    if num_frames <= required_num_frames:
                        # print(num_frames, required_num_frames)
                        cut_frame_array = append_frames(cut_frame_array, required_num_frames)
                        num_frames = cut_frame_array.shape[0]
                        # print(num_frames)

                    for batch_index in range(num_clips_per_video):
                        start_frame = random.randint(0, num_frames - window_size)
                        end_frame = start_frame + window_size

                        block_x[batch_index, index_in_batch, :, :, :, :] = np.array(
                            cut_frame_array[start_frame: end_frame, :, :, :])

                        basic_line = [0] * num_classes
                        basic_line[int(label) - 1] = 1
                        basic_label = basic_line
                        block_y[batch_index, index_in_batch, :] = np.array(basic_label)

                for batch_index in range(num_clips_per_video):
                    batch_start_time = time.time()

                    mini_batch_x = block_x[batch_index, :, :, :, :, :]
                    mini_batch_x = mini_batch_x / 255.0
                    mini_batch_y = block_y[batch_index, :, :]

                    perm = np.random.permutation(batch_size)
                    mini_batch_x = mini_batch_x[perm]
                    mini_batch_y = mini_batch_y[perm]

                    _optimizer, _cost, _prediction, _accuracy = sess.run([optimizer, cost, prediction, accuracy],
                                                                         feed_dict={x_inpuT: mini_batch_x,
                                                                                    y_inpuT: mini_batch_y})
                    epoch_loss += _cost
                    train_acc += _accuracy

                    num_batch_completed += 1
                    batch_end_time = time.time()

                    log1 = "\rEpoch: {}, " \
                           "batches completed: {}, " \
                           "time taken: {:.5f}, " \
                           "loss: {:.6f}, " \
                           "accuracy: {:.4f} \n". \
                        format(
                        epoch + 1,
                        num_batch_completed,
                        batch_end_time - batch_start_time,
                        epoch_loss / (batch_size * num_batch_completed),
                        _accuracy)

                    print(log1)
                    sys.stdout.flush()

            del block_x, block_y

            # validation loss
            val_loss = 0
            val_acc = 0
            val_num_batch_completed = 0

            num_clips_per_video = 1

            val_num_videos = num_val_videos
            val_num_blocks = int(val_num_videos / val_batch_size)

            val_block_x = np.zeros((num_clips_per_video, val_batch_size, depth, height, width, 3))
            val_block_y = np.zeros((num_clips_per_video, val_batch_size, num_classes))

            random.seed(23)
            print("Random seed fixed for validation.")

            for block_index in range(val_num_blocks):

                for index_in_batch, val_pkl_file in enumerate(
                        val_pkl_files_list[block_index * val_batch_size: (block_index + 1) * val_batch_size]):

                    with open(os.path.join(val_data_path, val_pkl_file), 'rb') as f:
                        frames_and_label = pickle.load(f)

                    cut_frame_array = frames_and_label[0]
                    label = frames_and_label[1]

                    num_frames = cut_frame_array.shape[0]
                    required_num_frames = int(window_size * ignore_factor)

                    if num_frames <= window_size:
                        cut_frame_array = append_frames(cut_frame_array, required_num_frames)
                        num_frames = cut_frame_array.shape[0]

                    for batch_index in range(num_clips_per_video):
                        start_frame = random.randint(0, num_frames - window_size)
                        end_frame = start_frame + window_size

                        val_block_x[batch_index, index_in_batch, :, :, :, :] = np.array(
                            cut_frame_array[start_frame: end_frame, :, :, :])

                        basic_line = [0] * num_classes
                        basic_line[int(label) - 1] = 1
                        basic_label = basic_line
                        val_block_y[batch_index, index_in_batch, :] = np.array(basic_label)

                for batch_index in range(num_clips_per_video):
                    val_batch_x = val_block_x[batch_index, :, :, :, :, :]
                    val_batch_x = val_batch_x / 255.0
                    val_batch_y = val_block_y[batch_index, :, :]

                    val_cost, val_batch_accuracy = sess.run([cost, accuracy],
                                                            feed_dict={x_inpuT: val_batch_x, y_inpuT: val_batch_y})

                    val_acc += val_batch_accuracy
                    val_loss += val_cost

                    val_num_batch_completed += 1

            del val_block_x, val_block_y

            epoch_loss = epoch_loss / (batch_size * num_batch_completed)
            train_acc = train_acc / num_batch_completed

            val_loss /= (val_batch_size * val_num_batch_completed)
            val_acc = val_acc / val_num_batch_completed

            epoch_end_time = time.time()

            log3 = "Epoch {} done; " \
                   "Time Taken: {:.4f}s; " \
                   "Train_loss: {:.6f}; " \
                   "Val_loss: {:.6f}; " \
                   "Train_acc: {:.4f}; " \
                   "Val_acc: {:.4f}; " \
                   "Train batches: {}; " \
                   "Val batches: {};\n". \
                format(epoch + 1, epoch_end_time - epoch_start_time, epoch_loss, val_loss, train_acc, val_acc,
                       num_batch_completed, val_num_batch_completed)

            print(log3)

            if save_loss_path is not None:
                file1 = open(save_loss_path, "a")
                file1.write(log3)
                file1.close()

            epoch_loss_list.append(epoch_loss)
            val_loss_list.append(val_loss)

            if save_model_path is not None:
                saver.save(sess, save_model_path)

        end_time = time.time()
        print('Time elapse: ', str(end_time - start_time))
        print(epoch_loss_list)

        if save_loss_path is not None:
            file1 = open(save_loss_path, "a")
            file1.write("Train Loss List: {} \n".format(str(epoch_loss_list)))
            file1.write("Val Loss List: {} \n".format(str(val_loss_list)))
            file1.close()


if __name__ == '__main__':

    np.random.seed(0)
    parser = argparse.ArgumentParser()

    parser.add_argument('-bs', action='store', dest='batch_size', type=int)
    parser.add_argument('-vbs', action='store', dest='val_batch_size', type=int)
    parser.add_argument('-lr', action='store', dest='learning_rate', type=float)
    parser.add_argument('-wd', action='store', dest='weight_decay', type=float, const=None)
    parser.add_argument('-e', action='store', dest='epochs', type=int)
    parser.add_argument('-nvv', action='store', dest='num_val_videos', type=int)
    parser.add_argument('-rc', action='store', dest='random_clips', type=int)
    parser.add_argument('-ws', action='store', dest='win_size', type=int)
    parser.add_argument('-slp', action='store', dest='save_loss_path', const=None)
    parser.add_argument('-smp', action='store', dest='save_model_path', const=None)
    parser.add_argument('-mn', action='store', dest='model_num', type=int)
    parser.add_argument('-vd', action='store', dest='visible_devices')
    parser.add_argument('-nd', action='store', dest='num_device', type=int)
    parser.add_argument('-if', action='store', dest='ign_fact', type=float, const=None)

    results = parser.parse_args()

    arg_batch_size = results.batch_size
    arg_val_batch_size = results.val_batch_size
    arg_lr = results.learning_rate
    arg_wd = results.weight_decay
    arg_epochs = results.epochs
    arg_num_val_videos = results.num_val_videos
    arg_random_clips = results.random_clips
    arg_win_size = results.win_size
    arg_save_loss_path = results.save_loss_path
    arg_save_model_path = results.save_model_path
    arg_model_num = results.model_num
    arg_visible_devices = results.visible_devices
    arg_num_device = results.num_device
    arg_ign_fact = results.ign_fact

    data_path = "/home/axp798/axp798gallinahome/data/jester/train_64/"
    val_data_path = "/home/axp798/axp798gallinahome/data/jester/valid_64/"

    ar_save_loss_path = None
    if arg_save_loss_path is not None:
        ar_save_loss_path = "/home/axp798/axp798gallinahome/Gesture-Detection/models/loss_jester/{}".format(
            arg_save_loss_path)

    ar_save_model_path = None
    if arg_save_model_path is not None:
        path = '/home/axp798/axp798gallinahome/Gesture-Detection/models/saved_models_jester/{}/'.format(
            arg_save_model_path)
        if not os.path.exists(path):
            os.mkdir(path)
        ar_save_model_path = path + "model"

    if ar_save_loss_path is not None:
        file1 = open(ar_save_loss_path, "w")
        file1.write("Params: {} \n".format(results))
        file1.write("Losses: \n")
        file1.close()

    depth = arg_win_size
    height = 64
    width = 64
    num_classes = 27

    os.environ['CUDA_VISIBLE_DEVICES'] = "{}".format(arg_visible_devices)
    os.environ['TF_CPP_MIN_LOG_LEVEL'] = "2"
    print(device_lib.list_local_devices())

    choose_device = "/device:GPU:{}".format(arg_num_device)

    with tf.device(choose_device):
        x_inpuT = tf.placeholder(tf.float32, shape=[arg_batch_size, depth, height, width, 3])
        y_inpuT = tf.placeholder(tf.float32, shape=[arg_batch_size, num_classes])

    train_neural_network(x_inpuT, y_inpuT, data_path, val_data_path,
                         save_loss_path=ar_save_loss_path,
                         save_model_path=ar_save_model_path,
                         batch_size=arg_batch_size,
                         learning_rate=arg_lr,
                         weight_decay=arg_wd,
                         epochs=arg_epochs,
                         val_batch_size=arg_val_batch_size,
                         which_model=arg_model_num,
                         num_val_videos=arg_num_val_videos,
                         random_clips=arg_random_clips,
                         win_size=arg_win_size,
                         ignore_factor=arg_ign_fact)
<file_sep>import os
import cv2
import time

videos_path = "/home/axp798/axp798gallinahome/ConGD/ConGD_phase_1/train"
store_frames = "/home/axp798/axp798gallinahome/store/train/frames"

script_start_time = time.time()

# print(os.listdir(videos_path)[270])

# dir = "246"
# curr_dir_path = os.path.join(videos_path, dir)
# print(curr_dir_path)
# print(os.listdir(curr_dir_path))

dir_count = 271
for dir in os.listdir(videos_path)[270:]:
    print("{}) {} started!".format(dir_count, dir))
    curr_dir_path = os.path.join(videos_path, dir)

    par_dir = os.path.join(store_frames, dir)
    if not os.path.exists(par_dir):
        os.mkdir(par_dir)

    dir_start_time = time.time()

    video_count = 1
    for video in os.listdir(curr_dir_path):
        if video[-5] == 'K':
            continue
        video_start_time = time.time()
        print("{}) {}-{} started!".format(video_count, dir, video))
        vidcap = cv2.VideoCapture(os.path.join(curr_dir_path, video))
        success, image = vidcap.read()
        count = 1
        # make directory // video = 00001.K.avi therefore video[:-6]
        video_frame_dir = os.path.join(store_frames, "{}/{}".format(dir, video[ : -6]))

        if not os.path.exists(video_frame_dir):
            os.mkdir(video_frame_dir)

        while success:
            # print("frames started!")
            cv2.imwrite(os.path.join(video_frame_dir, "frame{}.jpg".format(count)), image)     # save frame as JPEG file
            success, image = vidcap.read()
            # print('Read a new frame: ', success)
            count += 1

        video_end_time = time.time()
        print("{}) {}-{} is done in {} sec".format(video_count, dir, video, video_end_time - video_start_time))
        video_count += 1

    dir_end_time = time.time()
    print("{}) {} is done in {} sec".format(dir_count, dir, dir_end_time - dir_start_time))
    dir_count += 1

script_end_time = time.time()
print("Done in {} sec".format(script_end_time - script_start_time))
