Deep CNN-LSTM With Self-Attention Model for Human Activity Recognition Using Wearable Sensor

Human Activity Recognition (HAR) systems are devised for continuously observing human behavior - primarily in the fields of environmental compatibility, sports injury detection, senior care, rehabilitation, entertainment, and the surveillance in intelligent home settings. Inertial sensors, e.g., accelerometers, linear acceleration, and gyroscopes are frequently employed for this purpose, which are now compacted into smart devices, e.g., smartphones. Since the use of smartphones is so widespread now-a-days, activity data acquisition for the HAR systems is a pressing need. In this article, we have conducted the smartphone sensor-based raw data collection, namely H-Activity, using an Android-OS-based application for accelerometer, gyroscope, and linear acceleration. Furthermore, a hybrid deep learning model is proposed, coupling convolutional neural network and long-short term memory network (CNN-LSTM), empowered by the self-attention algorithm to enhance the predictive capabilities of the system. In addition to our collected dataset (H-Activity), the model has been evaluated with some benchmark datasets, e.g., MHEALTH, and UCI-HAR to demonstrate the comparative performance of our model. When compared to other models, the proposed model has an accuracy of 99.93% using our collected H-Activity data, and 98.76% and 93.11% using data from MHEALTH and UCI-HAR databases respectively, indicating its efficacy in recognizing human activity recognition. We hope that our developed model could be applicable in the clinical settings and collected data could be useful for further research.

I. INTRODUCTION
Human Activity Recognition (HAR) is a challenge that aims to forecast user activities based on device interactions. It aids people in their daily lives in a variety of ways. Human activity can be detected using two methods: video image recognition and wearable sensors. Through video systems, the camera is used to recognize human behavior. This strategy not only necessitates the installation of costly cameras and infrastructure, but it also creates issues because of the background, lighting, and scale circumstance that make movement detection difficult. In the following method, human activity identification relies on wearable sensors, in particular accelerometers, magnetometers, gyroscopes, linear acceleration and so forth convert motion into identified signals. It adds a new dimension to moving with fewer environmental constraints than the video-based method and provides the privacy of the user [1]. Wearable activity detectors, like the well-known Pedometer, are utilized in various healthcare applications for day-to-day fitness tracking [2], [3]. Despite this, the efficiency of those methods remains debatable [4], and much work is being done to enhance the contribution of inertial sensors for Human Activity Recognition (HAR). Although deep learning techniques for machine learning have recently taken a lot of consideration in the research community, deep learning norms are still underutilized for training time series of inertial sensor data for action detection [3], [5]- [8]. There have been numerous suggestions in the literature [3], [9]- [12] of traditional machine learning and profoundly trained methods for HAR using accelerometers over the last decade. In contrast, real-world HAR systems may fail to distinct to new members and/or situations, resulting in low computational efficiency in practical applications [13], [14]. The accuracy of activity-recognition algorithms can affect by a multitude of reasons, including (i) device position (for example, hand, pocket, or bag), or (ii) differences between sensor brands in terms of sensitivity range and sample frequency. While extensive explore has been completed on the effects of human traits on recognition accuracy, few studies have looked into the effects of position and device features [13], [15]- [18]. Lane et al. [17] developed a novel approach to incorporating human aspects. This method improves recognition accuracy by taking advantage of user similarity and weighting training data. Unfortunately, the researchers are unable to replicate the results obtained because the dataset which used for the experiment is not open access, and the authors' primary focus was on automatic annotation of inertial signals rather than the classification of subject activities. The proposed approach by Lane et al. [17] merits further investigation, which stems the research conducted in different disciplines.
Throughout this article, we explored the recognition of different activities i.e., standing/sitting, normal walking, running, and jogging on smartphone devices, and proposed an efficient and seamless combination of Convolutional Neural Network (CNN), Long Short Term Memory (an improvement of Recurrent Neural Network (RNN)) and a self-attention model for representing activity features. During the data collection phase, the use of smartphones is assumed to be unconstrained, and inertial sensors are of acceleration, gyroscope, linear acceleration types. In previous studies, features extracted from Micro-Electro-Mechanical Systems (MEMS) signals revealed that the gyroscope and accelerometer signals contain the most information about human motion because they measure kinematic motion indirectly [19]. CNN and LSTM for robust activity feature extraction are used for the time series convolutions features.
The above activities allow for supervised training to finally be carried out for the human activity recognition models. We have collected data with an app, namely 'sensor data collector' and we named our dataset is H − Activity dataset. H-Activity and two additional publicly available datasets, namely MHEALTH [20], [21] and UCI-HAR [22] were used for evaluating the proposed method. Smartphones were used to collect sensor data and stored for using in our study. The smartphone sensor data was obtained through experiments with ten participants and collect Standing/sitting, walking, jogging, and running data. These findings suggest that if sensor data is properly calibrated and advanced machine learning techniques like CNN, LSTM (Long Short Term Memory) and transformer learning architecture are used, a smartphone can be a powerful tool for recognizing human activities.
The contributions of this paper are listed as below: • Our system is the combination of deep learning with self-attention model and wearable sensor-based human activity recognition framework that utilizes various smartphone sensors.
• The proposed method uses only a three-axis accelerometer, gyroscope, and linear acceleration to provide reliable recognition performance, where no other sensors such as Global Positioning System (GPS) or a pressure sensor can perform likewise.
• Creating and evaluating a database (H-Activity): A total of ten subjects data is collected for four activities. Sensors on smartphones are used to collect data. The sensors were put in the user's right trouser pocket. Each activity was captured at a sampling rate of 10 Hz. Sensors capture a total of 9 attributes for each sample. Triaxial acceleration, gyroscope, and linear acceleration sensors were employed in the right pocket.
• We proposed the deep CNN-LSTM with self-attention model for activity classification problem using our own dataset H-Activity as well as two public datasets; MHEALTH and UCI HAR. Classification accuracy reached up to 99.93% for the H-Activity, and 98.76% and 93.11% for MHEALTH and UCI-HAR respectively.
• The suggested method is used to lessen the dependency on traditional Machine Learning ML) techniques that extract features in a handcrafted process.
This article has the following sections. The literature review, including related work, problem description, motivation, and feasibility analysis, as well as the advantages of the proposed method over existing methods, is described in Section II. Section III discusses the design of the system with H-Activity, MHEALTH and UCI-HAR datasets description, acquisition, and preprocess of the H-Activity dataset. The suggested human activity recognition framework is described in Section IV. Section VI discusses the results after describing the experiments and validation in Section V. Finally, Section VII and section VIII are for discussion and conclusion, respectively. VOLUME 10, 2022

II. LITERATURE REVIEW
This section describes existing work, the problem, and the advantages of the proposed work over existing techniques.
In the topic of human activity recognition, a significant amount of study has been conducted. The activity data collection methods usually varied of the sensing modality used. For the sake of this paper, we will exclusively explore strategies that make use of Smartphone sensor data. Smartphone sensor-based activity analysis has drawn the attention of researchers due to factors such as availability, affordability, and portability, as it eliminates the need for a sophisticated laboratory setup and expensive equipment.
Activity classification and feature extraction techniques have been studied in previous HAR. Deep learning is the rapidly emerging field which automates the aforementioned techniques. The deep learning technique, which employs numerous layers in the system, identifies ideal characteristics from raw data without the need for human interaction [31]. According to several research, this method can produce very accurate activity classification findings [32]- [34]. However, the application has limitations and challenges. To begin, training a deep learning model necessitates a large amount of data. Second, the model is typically treated as a black box, and the derived characteristics from the multi-layered approach may be difficult to understand [31], making algorithm improvement difficult. In [35] Long Short Term Memory has been used because of its nonlinear properties. The authors proposed a model for predicting green house climate change. Nevertheless, their used sensor collects incorrect data, but the model they proposed performs well with abnormal data. To recognize the upper limb gesture in a rehabilitation setting, the authors in [36] used a fully connected deep learning approach. They compare their model to various machine learning algorithms and show that the proposed fully connected neural network outperforms them in gesture recognition. In addition, the authors of [37] demonstrated that a category-aware gated recurrent unit model for the next POI category recommendation performs better than other baseline methods.
The researchers in [38] are working on human gait analysis for various clinical and pathological trails of patients with stroke, Parkinson's disease, old stage walking issues, and other neurological disorders. They employ several machine learning techniques that necessitate the services of a feature extraction expert. They proposed utilizing cellular automata to forecast human gait state and ELM to classify it. Human gait analysis is being considered by technology experts as a biometric identity verification method, multi-mode gesture generation, and the creation of human-like robot walking patterns. Semwal et al. [39] proposed an optimized feature for gait data categorization based on incremental feature analysis. This study relies entirely on skeleton characteristic data derived from human actions and deep learning models.Gupta et al. suggested a hybrid strategy for recognizing human walking behaviors using an ensemble learning method in [40].In [41], Bijalwan et al. presented a combination of wearable sensor-based and kinect sensor-based strategies for generating person stepping patterns, as well as constitutive models of the work.

B. PROBLEM DESCRIPTION, MOTIVATION, AND FEASIBILITY ANALYSIS
In this Fourth Industrial Revolution (4IR or Industry 4.0) era, the digital world has a plethora of data, such as mobile sensor data, security data, health data, and so on. Knowledge of artificial intelligence (AI), particularly deep learning (DL), is required to cognitively interpret these data in order to develop smart and automated applications such as elderly health issues and security systems, wearable and phone-based tracking systems, etc. In this field, deep learning algorithms of various types, such as convolutional neural network and long short term memory, are available. Deep learning is capable of efficiently analyzing large amounts of data. We present a comprehensive overview of deep learning algorithms that can be used to recognize human activity in this paper. Physical phenomena such as unbalance, stumbling, recurrent falls, staggering, and freezing in daily human movements are referred to as activity disorders. It can be caused by one of two factors: neurological or non-neurological. It's fairly common in adults in their forties and fifties, as well as those over the age of 80. Human activity recognition and classification, among other things, aid in the identification of neurological problem patients, hemiparetic patients, and the examination of sports-person activity patterns [38]. It is a sensor-based analysis technique that employs a variety of sensors to capture human activity or movement patterns. The use of smartphone sensors reduces the overall cost of the system.

C. ADVANTAGES OF THE SUGGESTED METHOD OVER EXISTING METHODS
The following are some of the advantages of the suggested approach over existing techniques: In contrast to present systems [42], which only recognize walking as an activity, the suggested concept recognizes four different activities. The proposed method is more accurate than the existing method. H-Activity is the name of the data set we have constructed.
The suggested study provides a basic framework for activity recognition as well as activities that are well suited for human activity analysis in clinical trials.

III. SYSTEM DESIGN
The objective of this research is to develop a model that can predict human activities including walking, standing/sitting, jogging, running, and so on. In the development of our human FIGURE 1. The schematic diagram of our proposed workflow. Raw data are firstly acquired from sensors. After preprocessing, segments of data are extracted (known as Segmentation) and a classifier is designed. Fine tuning is used to adjust the hyper-parameters. The classifier is then trained and evaluated using those features (known as Classification).
activity recognition system, a systematic workflow incorporating sensor engineering, data processing, and deep learning techniques is usually followed. This section discussed about the sensors, dataset and processing criteria that were used in the experiments. Figure 1 schematizes this approach, which consists of the following steps:

A. DESCRIPTION AND VALIDATION OF THE DATASET
Several wearable sensors-based datasets are available to detect human activity. However, the majority of data are gathered using a variety of sampling rates, sensor counts, sensor placements, and recorded activity counts. Using H-Activity as well as two commonly used datasets, including MHEALTH (mobile health) [20], [21] and UCI-HAR dataset [22], the suggested technique and current approaches provided in the study were experimented and verified. These datasets are organized at various sample rates, the number of sensors and the number of activities recorded. A few of these datasets are well-balanced, while others are significantly unbalanced. We have compared a classification method across different datasets. Although these datasets have various frequencies and activities, our goal is to establish the novelty of the same LSTM-CNN classifier that can produce satisfactory results in different circumstances. One way to compare the data is to make them similar, as done in [43]. Though we can only change the data of proposed datasets, since we have the complete data for each activity, we can divide them using separate methods such as transformation. However, it is not easily attainable for the remaining datasets.
A brief summary of the H-Activity, MHEALTH, and UCI-HAR that were used in the proposed study is discussed in Table 1. Figure 2 shows the sensor placement on subject's body. The relevant sections provide a quick overview of the mentioned datasets:

1) H-ACTIVITY
The H-Activity dataset was gathered from ten volunteers between the ages of 20 and 30, who kept their smartphones in their right trouser pockets. All of the volunteers were participating in 4 different physical activities. For instance: sitting/standing, walking, jogging, and running.

2) PUBLICLY AVAILABLE DATA: MHEALTH AND UCI-HAR
To validate our collected dataset, we use the previously available MHEALTh and UCI-HAR datasets as a benchmark for making comparisons with our collected data.
Sensors placed at the left, chest, and right handles of 10 volunteers collected body signals and emergent indicators (swing rate, acceleration, magnet field direction), while performing 12 physical activities, such as Sitting and relaxing Dataset UCI-HAR is made up of data collected by 30 people. The ages of the participants range from 19 to 48 years old. Walking (Walk), going upstairs (Up)/downstairs (Down), sitting (Sit), standing (Std), and lying (Lay) are among the six actions performed by the participants. A Samsung Galaxy S II smartphone was attached to the waist for collecting accelerometer and gyroscope data. Fixed segmentation of the data with 50% overlap was employed for this data collection. A butterworth low-pass filter was employed to distinguish the components of gravitation and body motion of the sensor data into the acceleration of body and gravity. The remaining 30% of the participants were utilized for testing and the rest 70% were employed for training. During testing, this data splitting was leveraged to generate new, previously unseen data. This dataset contains 10,299 samples.

B. ACQUISITION AND PREPROCESSING OF H-ACTIVITY DATASET
In order to collect the time-series H-Activity data, we used an Android-OS-based application, namely Sensors Data  Collector to select the inertial sensors, i.e., acceleration, gyroscope, and linear acceleration. Further details of this application can be found in reference [44]. This data collection app has a lot of customization options available and the user has the option of selecting among Sitting/Standing, Walking, Jogging, and Running. In addition, the user can customize the sensors readings (which are by default set to one hour) and the data capture speed (which is set to fast). Furthermore, participants have the option of selecting which sensors will collect data, for example, a gyroscope, an accelerometer or a linear acceleration.
The data collection program, 'Sensors Data Collector' shown in Figure 3(a) is used to gather information from 3 sensors: accelerometer, gyroscope, and linear acceleration during these activities. The way in which machine learning algorithms function was not to read a huge quantity of data at once, so that each entry in the H-Activity dataset was done with 10-sec segments, as recommended in [45]. The app captured raw data from each line of sensors, each of which has 3 axes: X, Y, and Z. The major problem in this section is that the Android platform does not allow data from a sensor to be read at a particular moment. In practice, the data can only be read when a shift in the sensor reading is detected. Another issue was noise filtering. The data was not subjected to noise cancellation. To avoid inaccurate labeling, topics have to come to a complete halt, waiting for a couple of seconds before proceeding to the next [46] before each action. Accelerometer, gyroscopic and linear data from the start and stop periods of the activity were used as the name of the activity. The sampling rate was 10Hz, which means one sample was taken every 1 second. A total of 48,920 threedimensional acceleration, gyroscope, and linear acceleration samples were collected.
Both an accelerometer and a linear acceleration sensor calculate the force applied to a system in 'meters per second' on all 3 physical axes (X, Y, and Z). The difference between the two is that the accelerometer includes the force of gravity, whereas the linear acceleration does not. The data was unbalanced, particularly when walking normally, for which 'class balancing (i.e., Under-sampling or oversampling) the data set  was required. Table 2 shows that during normal walking, the linear acceleration and gyroscope only yielded total 12,233 and 12,247 data, respectively, while the accelerometer gave 77,867 data.
In the proposed method, the nine features, namely accelerometer, gyroscope, and linear acceleration in the x,y, and z axes, are used as input to the learning algorithm. Each sample is labeled as the data is processed. Under and Over Sampling, Linear interpolation and data segmentation are performed, followed by one-hot vectorization.

1) UNDER-SAMPLING AND OVER-SAMPLING
Each of the three sensor data represents the same time duration, though it contains a different amount of data for a similar time stamp. Since each sensor has a different sampling rate and vendors, the number of data points varies for accelerometer, linear acceleration and gyroscope. To our observation, an accelerometer generates more data than a gyroscope and linear acceleration. One solution to the issue of class imbalance is to resample the training dataset at random. Under-sampling (deleting samples from the majority class) and over-sampling (duplicating instances from the minority class) are the two key ways to randomly resample an imbalanced dataset. Under-sampling is generally beneficial, while random oversampling is not. SMOTE (Synthetic Minority Oversampling Technique) is a widely used oversampling technique that was developed to enhance random oversampling. To overcome the problem of data imbalance, it was then over-sampled using minimum and maximum values. Table 3 illustrates the distribution of data with different activities after sampling.

2) INERTIAL SENSORS IN THE SMARTPHONE
An Inertial Measurement Unit (IMU) is a digital gadget that assesses or calculates and states a specific force, angular rate, and, in some cases, magnetic fields surrounding the body using an accelerometer and gyroscope combination. Currently, most smartphones support all of these sensors. The inertial dynamics are measured in three directions along the x, y, and z axes, as shown in Figure 3(b). The three-way accelerations change linearly with the smartphone velocity in 3D space and depict the movement of smartphone users. Smartphone's angular speed is collected by the three-axis gyroscope, as it spins in space and can also be used to define the user movement. Smartphones of various brands such as Samsung, Xiaomi, and Huawei, all running the Android operating system, have been used in this study. The smartphone is turned on when a user walks and the spinning occurs according to the user's movement. These data are each deemed unique so that we can collect them for Guide Dynamics as source data. The smartphone can handle the accelerometer, gyroscope and linear acceleration with hardware synchronization. If intermediate data are recorded via the Android interface, it can affect a certain level of synchronization. But it is bearable and has minimal impact on our system of recognition of activity. In this investigation, the accelerometer is employed for the x-axis, y-axis and z-axis three times. The body movement and gravity combine each time series with linear acceleration. We're mostly interested in gathering data on three types of oriental motions (horizontal, vertical, and backward/forward with relation to the x, y, and z-axes). Figure 3(b) shows an example of tri-axial data illustrating a user conducting walking.

3) LINEAR INTERPOLATION
We acquired the H-Activity dataset practically by using the smartphone's sensors. The smartphone is placed in the right pockets of subjects. As a result, some used data has been lost during the compilation process, which is typically represented by NaN or 0. This problem can be overcome by using the linear interpolation algorithm and filling the missing value [47]. We used the interpolation algorithm to deal with the problem.

4) SEGMENTATION
In this article, a model for the recognition of human activity was introduced. A data sequence is used to create the model input. The sequence is derived from a raw sensor's short time series data and data was collected continuously throughout the data collection process. To retain the temporal relationship between data points, a sliding operation for segmentation was used, which had the size of 10 in length. Each window of readings is a 9-feature data point and the size of the data is 6 MB.

5) CLASS RELABELING AND ONE-HOT ENCODING
The output labels have also been converted to one-hot encoded labels. As discussed in the H-Activity dataset description section, we coded our activity windows into four unique labels. These are: '0' for sedentary (stand + sitting), '1' for normal walking, '2' for jogging, and '3' for running. The steps of pre-processing are also summarized in Figure 4.

IV. PROPOSED HUMAN ACTIVITY RECOGNITION ARCHITECTURE
The proposed HAR model architecture, which is based on CNN-LSTM with a self-attention model, is used to classify smartphone users' activities. In this study, the framework allows for the collection of sensor data from a smartphone sensor. Figure 4 depicts the structure of the proposed CNN-LSTM with the self-attention model. It consists of nineteen layers. Preprocessing of raw sensor data is described in sections III(B).
The pre-processed input data is first fed into a 16-filter convolutional neural network, which is then followed by a batch normalization layer and a dropout layer with a 20% rate. The output is then fed into a 64-neuron two-layer LSTM, which is commonly used to extract temporal information. Second, the output of the LSTM layer has been transmitted into the self-attention layer. The attention layer is primarily used to focus on a particular network layer. The second LSTM, attention, and dropout layer employed the same number of neurons and dropout rate as the previous step. The activation is accomplished through the use of a'sigmoid' feature. Finally, an output layer (a dense layer with a'softmax' classifier) is used to obtain the model's output. The Adam optimizer outperformed the other three types of optimizers (i.e., Adagrad, RMSprop, and SGD). All of the parameters, as well as binary and categorical cross-entropy, were tested on a trial-and-error basis, and the best one was chosen.

A. NEURAL NETWORK FOR HUMAN ACTIVITY RECOGNITION: MODEL IMPLEMENTATION
The human activity recognition network, as shown in Figure 4, is made up of a CNN, LSTM, attention, and a dense network. The activities of the subject are recognized using a dense network, which performs the function of a classifier using the residual concatenation for classification, followed by CNN, LSTM and the attention model. Figure 4 depicts the proposed CNN-LSTM framework with self-attention model, which employs CNN layers to automatically extort characteristics from facts, as well as combined with LSTMs and an attention layer to aid sequence forecasting. CNN-LSTMs with self-attention are employed in the solution of visual time series forecasting problems and the generation of textual records from image series. This construction is relevant for problems that require temporal output generation or that involve temporal input structures. This paper proposes a deep CNN-LSTM with self-attention model to improve recognition performance.

1) CONVOLUTIONAL LAYER
where a i,j is the respective activation, f is the non-linear activation function, W m,n represents the weight matrix of convolution kernel as m×n, X i+m,j+n denotes the upper neuron activation connected to the neuron (i, j), and b is the bias term. In our experiment, rectified linear units (ReLU) are used by convolutional layers in the calculation of the feature maps, where the non-linear function is denoted like the following: In general, it is examined that the more convolution kernels are used, the more hidden features of the input samples could be extracted [48]. The CNN-LSTM with self-attention

2) LSTM NETWORK STRUCTURE
Nowadays, LSTM networks [49] perform admirably across a wide range of temporal schemes. The LSTM is a type of Recurrent Neural Network (RNN) that is growing in popularity. RNNs can estimate the present time output relied on previous knowledge in the DL approach. However, due to the disintegrating gradient problem, [50] states that RNN systems can only identify data for a short period of time. Gradients will be submerged if they are not allowed to flow deeply while using the deep learning back-propagation strategy. To address the challenge of long-term dependency, [51] proposed a novel neuron to the RNN group called LSTM.
To extract the temporal aspects in the sequence data more effectively, the input data is first passed through two-layer of LSTM in this paper. There are 64 memory cells in the LSTM layer. The following formula is used to manipulate the action of each LSTM unit by sending various inputs to different gates, such as input, gates, and output gates.
where h t and h t−1 signify activation at time t and t-1, correspondingly, w i,h is the input-hidden layer weight matrix, w h,h is the hidden-hidden layer weight matrix, b is the bias, and a non linear activation function is symbolized by the σ .

V. EXPERIMENTS AND EVALUATION A. EXPERIMENTAL SETUP
For a recurrent network, the size of the network depends on the availability of the GPU memory that is used and on the users' duration of the training. The size of the GPU RAM should be larger to train a larger CNN and RNN. As an example, LeNet5 training requiring 1GB of GPU RAM can be considered. In this article, the experiment is run on a computer with an Intel Core i5 processor, 8 GB of RAM. In terms of software, the Google COLAB server is used to compile the experimental analyses. We have used Tensorflow as a python library for conducting deep neural network training, tensor operations and parameter inference with automatic differentiation. Otherwise, for other array operations, Numpy is used, while matplotlib and seaborn are used for data visualization.

B. EVALUATION MEASURES
When collecting data on human action in natural settings, class-imbalances in data categories are common [52], and in this article, our data sets are not different from that aspect. The results will achieve high accuracy if the classifier predicts each instance as a majority class and uses the overall classification accuracy to assess the model output. Therefore, the overall classification accuracy is not a good indicator of the model performance evaluation. The F-measure (F1 score) considers both false positives and false negatives, and it incorporates two metrics based on the total number of correctly recognized samples, known as 'precision' and 'recall'. Next, we briefly describe the evaluation criteria that are used in this study:

1) PRECISION
The ratio of correctly predicted positive observations to total predicted positive observations is known as precision.
In Eq 4 T P and F P denote true positive and false positive respectively.

2) RECALL(SENSITIVITY)
The ratio of correctly expected positive observations to all observations in the actual class is known as recall.
Recall(Sensitivity) = T P T P + F N (5) In Eq 5, the T P and F P denotes true positive and false positive, respectfully.

3) F1 SCORE
It is a harmonic average of the 'Precision' and 'Recall' values. Therefore, this score considers both false positives and false negatives to higher in order to get a higher F1-score. Although it is not as intuitive as accuracy, F1 is generally more useful than accuracy, particularly when the class distribution is uneven.

VI. EXPERIMENTAL RESULT ANALYSIS
To obtain the final model with the best results, four different models were designed and tested. As a result, a number of experiments have been carried out in order to fine-tune the parameters. Different input and output sizes, as well as the Convolutional layer, LSTM layer, attention layer, number of dense layers, and dense layer parameters, all were taken into account during this model selection process. Each model has been trained for a total of 150 epochs. Normally, as training epochs increase, DL models accuracy increases and loss (of the cost-function) decreases. Since the proposed model has been converged in a steady accuracy after 150 epochs, only 150 epochs have been used in both for training and validation. A detailed description of the parameter settings for four different models and the optimized model selection process is outlined below.

A. HYPER PARAMETER SELECTION
For selecting the best model with optimized hyper-parameter settings, we've designed and tested four different architectures. In the first model, namely M1, we used a 10 × 9 input matrix with two LSTM, two attention models, and two batch normalization, four dropouts, and three dense layers. Then, for each of the two LSTM layers, a total of 64 neurons were used. Furthermore, the output of the last LSTM layer becomes 128, which is the input of two different dense models. These dense networks are the combination of dense, batch normalization, and dropout layer, where all the inputs and outputs of the first dense model have a total of 64 neurons and in the second dense model, only the first input is 128, and the rest of the layers have 64 neurons as input and output. In the second model, namely M2, we have used a 10 × 9 input matrix with three LSTM, batch normalization, dropout, and dense layer. Then, for each of the three LSTM layers, a total of 32 neurons were used. Furthermore, the output of the last LSTM layer becomes 128 (the input data of the first batch normalization layer). The inputs and outputs of the second and third batch normalization layers are 256×256 and 128 × 128, respectively.
Next, in the third model, namely M3, We used a 10 × 9 input matrix with two LSTM, three batch normalization, three dropouts, and three dense layers for the third model, namely M3. Then, a total of 32 neurons were used for each of the two LSTM layers. Furthermore, the output of the last LSTM layer becomes 128 (the input data of the first batch normalization layer). The inputs and outputs of the second and third batch normalization layers are 256 × 256 and 128 × 128, respectively.
And finally, in the fourth model, namely M4, which is also our proposed model, we have used a 10 × 9 input matrix with one CNN model, three LSTM, two attention models, and three batch normalization, five dropouts, and two dense layers. Then, for each of the first two LSTM layers, 64 neurons were used, and the final LSTM layer had 128 neurons. Furthermore, the output of the final LSTM layer was 384, and served as the input to two separate dense models. These dense networks were a mixture of dense, batch normalization, and dropout layers, with all inputs and outputs in the first dense model, totalling 320 neurons, and only the first input in the second dense model totalling 512, with the rest of the layers having 64 neurons as input and output. The dropout layer is used to drop the layers that aren't required. The number of trainable parameters for these models was 152 902, 156 292, 197 508, and 633,188 respectively.
For each of these four models, the final output was a one-dimensional vector after applying the pixel operations of 320 × 4, 128 × 4, 128 × 4, and 512 × 4 dimensions, respectively. Model architectures of these four models are outlined in Table 4. The comparative performances of these four models, regarding their training and validation accuracy, Area Under the ROC Curve (AUC), F1 score and Loss are shown in Figure 5 and Figure 6, respectively.
While finding out the best performing one among these four models, we observed the above performances, where M4 demonstrated the best performance among all others overall. Thus M4 was chosen as the final model for any further analysis. Moreover, M4's optimal hyper-parameter settings are described in Table 5, which were trained for a total of 150 epochs and corresponding Loss, recall, precision and F1 score have been determined for each epoch of training and testing. Finally, the average validation accuracy for all epochs was achieved as 0.991452719, while the loss was 0.043110911, and F1 score was 0.99182258.
When predictions were made for the test data set, we presented the confusion matrix plot in Table 678 to put our model performance in perspective. The actual class (Target Class) is defined by the rows in the confusion matrix, while the predicted class is indicated by the columns (Output Class). The uncertainty matrix diagonal cells correspond to correctly classified observations (TP, True positives and TN, True Negatives). In our research data set, there are 4,888 instances of correctly classified for different activities. The findings that were incorrectly labelled (FP, False Positives and FN, False Negatives) are represented by the off-diagonal cells.
All networks are trained with 150 training epochs, where a different number of epochs were used, and we observed that after 150 epoch, the results were repeated, so 150 epoch was finalized. After applying numerous amounts of data as an experiment, we use 25% of the whole dataset for testing.

B. MODEL EFFICIENCY AS A RESULT OF HYPER-PARAMETERS: INFLUENCE OF THE OPTIMIZER
The optimizer adjusts and analyses network settings impacting model training and performance to approximate the optimal benefit while decreasing the loss function. As a result, selecting an appropriate optimizer for deep model training is critical. Various well-known optimizers, such as Adam, Adagrad, SGD, and RMSprop, were analytically examined, as shown in Figure 7. It is observed that the Adam optimizer appears to have the greatest effect on model efficiency, with the gradient descent curve fluctuation being the most stable. Hence, when training the CNN-LSTM with self-attention model, Adam was used as the optimizer.

C. EVALUATION ON H-ACTIVITY, MHEALTH, AND UCI-HAR DATASETS
H-Activity, MHEALTH, and UCI-HARt were utilized for testing to fully validate the performance of the suggested model. Tables 6, 7 and 8 demonstrate the confusion matrices of classification that were produced when the H-Activity, MHEALTH, and UCI-HAR datasets were predicted for the model. There were 4888 occurrences properly categorized for the H-Activity data set and the total accuracy was 99.93%. The precision and recall were between 99% -100%. Between jogging and running, there was rather weak differentiation. The principal reason is that from the point of view of motion sensors, the two activities are nearly identical. When the model was subjected to the test set including around 330978 new instances, the overall accuracy of the dataset MHEALTH (it contains 12 activities) reached 98.76%. There were 2714 properly categorized occurrences for the UCI-HAR data set, with total accuracy reaching 93.11%. Between sitting and standing there was rather low VOLUME 10, 2022   discrimination like H-Activity dataset. Recall and accuracy were in the 80% by 100% range. The major explanation for this might be that the two activities are comparable in terms of motion sensors. Deeper information is difficult to extract with simple acceleration, linear acceleration, and angular velocity data.  Proposed deep CNN-LSTM with self-attention model was compared with LSTM-CNN from Lyu et al. [53] and CNN-LSTM [54] under the same experimental scenario in order to further verify the model performance. All results were carefully verified to ensure that the results of the comparison were fair and uniform. The evaluation results for the above profound models are shown in Table 9. The deep CNN-LSTM with self-attention has significantly increased by about 3% for the MHEALTH dataset, compared to LSTM-CNN model of Lyu et al. It can also be observed that CNN-LSTM with self-attention outperforms the CNN-LSTM, CNN, and Res-LSTM model, proposed by Mutegeki et al. [55], Cruciani et al. [56], Ullah et al. [57], and Yu Zhao et al. [58] on the UCI-HAR by nearly 1%.

VII. DISCUSSION
In this section we conducted a series of experiments to extensively assess the effectiveness of the model described above in order to confirm that the Deep CNN-LSTM with Self-Attention model would perform as expected. We used the three datasets discussed in Section III above to run various experiments on the models, and the results are shown in the following sections.
Initially, data has been categorized into different classes to make classification easier, i.e., sitting, walking, jogging, running, standing and so on. Different deep learning architectures were designed and tested to find out the best fitted model for the recognition. A parallel dense network has been utilized before output parameters. This parallel processing allows  an optimal path to be chosen in the hidden layer. Residual connection from the previous layer also prevents vanishing gradient problems [52]. The findings of the study clearly state that various forms of activity can be easily identified.
We demonstrate how convolution processes are robust enough to be applied directly to raw sensor data to extract features that surpass earlier results on the subject within a deep framework. The use of CNNs has the advantage of avoiding hand-crafted or heuristic features, which reduces engineering bias. This is especially crucial when using activity recognition approaches in domains with more complicated activities or open-ended scenarios, where classifiers must adapt to a changing number of classes.
We used the deep CNN-LSTM with self-attention model, which is a novel model to be used in the research problems like this. Three different models (i.e., M1, M2, and M3)  have been compared with our chosen model (i.e., M4). In contrast to these three models, i.e.,sequential LSTM layers (LSTM-CNN) before Convolution layer [48], sequential Convolution and dropout layers (CNN-LSTM) before LSTM layer [59], parallel LSTM layers (Parrallel LSTM-CNN) with Convolution layer [60], our chosen model achieved the best testing result of 99.93% for H-Activity dataset and 98.76% and 93.11% for MHEALTH and UCI-HAR dataset respectively. Figure 8 compares test F1-Score, Accuracy and AUC among these models. Note, the comparison using final statistical measures with other existing approaches was not conducted since the data set and experimental settings differ from our article.
Hence, all of the above-mentioned models [48], [59], [60] were implemented with exact parameters described in the respective literature except the input and output layer to match our dataset. F1-Score, Accuracy and AUC of best performing Attention CNN-LSTM model (Model4) from the previous sub-section has been added in the figure 8 for contrast. Although every model has achieved sufficient performance in testing data, CNN-LSTM with self-attention model has better results than other models. Thus the significance of the deep CNN-LSTM with self-attention model has been proved for the human activity recognition approach.

VIII. CONCLUSION
We have evaluated a Deep CNN-LSTM with Self-Attention model using Wearable Sensor for the classification of daily activities. We presented this network model using raw accelerometers, gyroscopes and linear Acceleration Data of a smartphone for the input. We also used two benchmark datasets, i.e MHEALTH and UCI-HAR to demonstrate the robustness of our proposed model and get accuracy 98.76% and 93.11% for MHEALTH and UCI-HAR datasets respectively. Our research shows how the automated feature engine in CNN and LSTMs can efficiently extract these characteristics. In a four-class activity recognition scenario with a ten voluntary personalized data set, i.e. H-Activity, the presented model achieved an accuracy of 99.93%. The proposed model demonstrated greater solidity and was more likely than models using statistical machine learning techniques to detect human activity. In the future, we will first continue to strengthen our dataset by adding more participants and adjusting our network structure. Our future research will concentrate on real-time classification of elderly health issues and security systems. We will also focus on the development of wearable and phone based tracking system. Thus, we believe that this developed framework could be applicable in the clinical setting and collected data could be useful for further research.


Multiple Transform Selection Concept Modeling and Implementation Using Dynamic and Parameterized Dataflow Graphs

The new video coding standard, Versatile Video Coding (VVC), released by the end of 2020 has increased the coding complexity both at encoder and decoder sides. This complexity increase is due to several coding tools proposed to enhance the coding efficiency. One of these tools is the Multiple Transform Selection (MTS) concept, a new approach for the transform unit. This paper aims at providing a new optimization of the MTS based on dataflow modeling. The proposed approach takes benefit of the different parallelism levels of the MTS in order to create an optimized multicore implementation. Also, this paper study the impact of the dataflow model granularity and the dynamic reconfiguration on the implementation efficiency on x86 multicore architectures. The PREESM tool is used in this study to develop the proposed dataflow models and for the granularity analysis. The dynamic reconfiguration study is here performed using the SPIDER runtime optimized for the multicore execution of applications modeled using Parameterized and Interfaced Synchronous Dataflow (PiSDF) dataflow graphs. Two architectures were used in this work: an x86 architecture with 4 cores and an x86 architecture with 24 cores. The results show that the SPIDER overhead time is almost negligible (0.05%) compared to the execution time of the application. Furthermore, a speed-up of 3.9 and up to 22 for all block sizes was achieved using a 4-core and 24-core machine, respectively.

Introduction
Since 1988, the appearance of the first MPEG-1 standard, a variety of MPEG video coding standards has been developed. The process of standardization has always focused on providing efficient coding tools for a wide and easy deployment. The increase of video traffic, which is expected to reach 82% of global Internet traffic by 2022, has led to the appearance of a new video coding standard called Versatile Video Coding (VVC) [1]. This latter saves 40% of the bitrate compared to the previous standard High Efficiency Video Coding (HEVC) for the same visual quality [2,3]. VVC offers many advantages dealing with higher resolutions, larger color gamut and a variety of different applications. However, the decoding complexity has been doubled compared to HEVC. This increase in complexity is due to the adoption of new efficient coding tools. One of this coding tools is the Multiple Transform Selection (MTS) concept. The MTS concept is based on three transform types (DCT type II (DCT-II), DCT type VIII (DCT-VIII), and DST type VII (DST-VII)). The encoder executes the three types of transformation and then applies the one resulting with the lower rate distortion for each block to encode. The MTS concept has significantly improved the compression ratio, but it raises the computational complexity for the encoder and the decoder as well [4]. Unfortunately, due to the additional computing complexity, both the execution time and the hardware cost are increased. These facts have led several researchers to propose optimizations for the MTS concept. The majority of the optimizations have focused on reducing the number of required operations (addition, multiplication) or on deducing one transform type from another. This paper introduces an extended version of [5]. Both papers aim at providing a method for optimizing the MTS concept based on dataflow modeling and studying its efficiency. Dataflow modeling has proven its efficiency many times in the past, especially in the field of signal and image processing. Dataflow modeling is a powerful tool to study data dependencies and to reveal different levels of parallelism. Whereas [5] investigates the use of static dataflow models to optimize and evaluate the MTS at compile time, this paper extends the results using dynamic dataflow models and extends the analysis at runtime. The tool used to create the dataflow models for the MTS concept is called Parallel and Real-time Embedded Executives Scheduling Method (PREESM). PREESM is also used to analyse the graphs and to automatically generate an optimized multicore algorithm for the MTS. PREESM allows to implement a multicore algorithm characterized by an optimal use of available computing resources (memory, computing units) in days whereas traditional methods take months. Dataflow modeling has been used on previous video coding standards, for example in [6,7] a proposal of dataflow models for Mpeg-4 and Advanced Video Coding (AVC) using SynDEx and Orcc has been introduced. To the best of our knowledge, this is the first dataflow model for the MTS concept presented in the VVC standard. The proposed model takes advantage of PREESM features such as memory optimization and automatic pipelining. Three points summarize the contributions of this work: 1. The proposition of an efficient Interface-Based Synchronous Dataflow (IBSDF) graph for the MTS concept. 2. The transformation of the IBSDF graph into a Parameterized and Interfaced Synchronous Dataflow (PiSDF) graph in order to study the different challenges of dynamic behaviour on the application performance, also details the extra benefits that a PiSDF graph provides compared to an IBSDF graph. 3. The generation of a multicore optimized algorithm for the MTS concept.
The rest of this paper is organized as follows. Section 2 presents a background on the design of the MTS concept and the state of the art of its hardware and software optimizations. It also introduces the importance of using dataflow modeling in embedded systems and gives an overview of the PREESM framework. Section 3 studies the impact of granularity on application performance while using an x86 architecture and based on the IBSDF graph. A discussion on the parallelization possibilities of the MTS concept was also provided. In Sect. 4, a proposal for a PiSDF graph is introduced. Moreover, a study of its efficiency using x86 architecture is detailed. Finally, Sect. 5 concludes the paper.

Background
This section encompasses the different optimizations made for the MTS concept in a first part and, in a second part, it introduces the benefits of dataflow modeling on the application performance, as well as the different dataflow models (IBSDF and PiSDF) and the different tools used to create these models.

Multiple Transform Selection Concept
The transform unit has been the focus of different researchers because of the computing complexity it contains. Several optimizations have been proposed, both in terms of software and hardware. This section concentrates on the optimizations of the MTS concept. The MTS has been introduced in the VVC standard [8]. The reason to use several types of transforms is based on an awareness that using a single transform to model the different statistical variations that might be contained in an intra prediction residual block is not efficient [9]. MTS allows the VVC encoder to choose the transform that minimizes the rate distortion among predefined trigonometric transforms, including DCT-II, DCT-VIII and DST-VII. The MTS concept is based on three features which are the butterfly decomposition for DCT-II, the derivation of DCT-VIII from DST-VII and vice versa, and the zeroingout of high frequency coefficients for block of size 64 × 64 for DCT-II and of size 32 × 32 for DST-VII and DCT-VIII. The majority of the improvements made to the MTS concept, whether hardware or software, are based on minimizing the computational complexity (addition, multiplications) or on the derivation of one transform type from another. In [10], a link between the two types of transformation DCT-II and DST-VII was found by Reznik. Authors in [11], provides a detailed study of two approximations; the first for DCT-VIII and the second for DST-VII. The first approximation explored the link between DCT-VIII and DST-VII. In fact, DCT-VIII can be inferred from the DST-VII without causing additional computational complexity, based on the vector reflection matrix Γ and the sign change matrix Λ as depicted in the following Eq. (1) where C 8 and S 7 present respectively the transforms coefficient matrices of DCT-VIII and DST-VII. The vector reflection matrix Γ and the sign change matrix Λ are derived using the Eqs. (2) and (3), respectively. (1) with i, j ∈ {1, 2, … , N} and N is the transform size. The second approximation is based on the link between the DST-VII and the DCT-II. In fact, the estimated DST-VII can be obtained using DCT-II as the expression 4 shows.
where S 7 = Λ ⋅ C T 2 is equivalent to the DST-III transform, the matrices Λ and Γ are calculated by the Eqs. (2) and (3), respectively, and A being a sparse banded matrix. Due to this approximation, the reduction of computational complexity is ensured as it is based on the DCT-II algorithm which is known by fast computation. Actually, DCT-II forms the base of several approximations due to the symmetry and recursion properties that it offers [12,13]. In [14], a method based on the factorization of fast implementation algorithms was adopted in order to minimize the number of required operations by the matrix multiplication. There are three properties of the DST-VII. These properties make it possible to factorize many multiplications, which have the same absolute value coefficient in the same baseline, in a single multiplication operation. In [15] a 2D implementation of the HEVC DCT transformation has been introduced. The proposed design takes advantage of multiple hardware resources, such as multipliers, DSP blocks and memory blocks, to reduce the logic utilization. This reconfigurable architecture can handle blocks of size up to 32 × 32 . The proposed architecture has been implemented in diverse Field Programmable Gate Arrays (FPGA) platforms. The result showed that the design could support 4Kp30 video encoding with a reduced hardware cost. In [16] a pipelined 1D hardware implementation of the Adaptive Multiple Transform (AMT) for all block sizes up to 32 × 32 have been proposed. The design has been implemented on various FPGA chips based on several Read Only Memory (ROM) in order to store the matrices of transform coefficients. Despite of its low hardware cost, the design can only support 1D AMT design or the transform process is based on 2D operations which can be more complex. In [17], a hardware implementation was proposed by investigating two hardware methods with a fixed throughput of 8 pixels/cycle. The first employs separate data paths and the second takes into account reconfigurable data paths for all 1D transforms. Although, the efficiency of the proposed implementation is limited to a block size of 8 × 8 . However, larger block size transforms are more complex and require more resources. In [18], a hardware implementation of 2D has been proposed but it does not support blocks of size 64 × 64 . In [19] another 2D hardware implementations for DCT-VIII and DST-VII has been introduced. Despite being the first implementation that takes advantage of asymmetric blocks, it does not support the DCT-II transformation. There are some approximations for the MTS transforms that were introduced in [20][21][22], and [23]. The proposed implementations aim at using the transform DCT-II to approximate DST-VII and DCT-VIII. All optimizations use the same concept, except that the contribution of [23] offers less complexity with almost the same coding efficiency. In this paper a new software optimization for the MTS concept based on dataflow modeling is introduced. Unlike the state of the art optimizations, the proposed optimization does not focus on reducing the number of computing operations neither referring one transform type from another. Moreover, it considers all transform sizes. It aims at revealing the independency between tasks and explore parallelism in order to speed-up the application performance.

Dataflow Modeling
Dataflow modeling is a powerful tool to capture numerous data dependencies and to explore parallelism between cores, thanks to the additional insight into the application it provides. Besides, dataflow modeling allows targeting different architectures with the same dataflow model. Dataflow is used extensively in Digital Signal Processor (DSP) applications and it has proven to be effective in facilitating the exploration of the various stages of parallelism [24]. In addition, the dataflow modeling is commonly used in the design of Multiprocessor Systems-on-Chip (MPSoC) [25,26]. The complexity of the video coding standards has increased in the last decades in order to increase the video quality and the user experience. Parallel architectures provide suitable solutions to cope with this complexity at the cost of longer software developments. The reduction of the software development effort demands the use of a software tool capable of automatically generating an optimized multicore implementations. In this paper, we evaluate software tools based on dataflow models.
The tool used for creating the dataflow model is called PREESM. PREESM is an Eclipse-based open-source framework that generates C code for embedded multicore systems from dataflow models of the application. Other tools have been employed to create a dataflow model for some previous video coding standards, such as Open RVC-CAL Compiler (ORCC) and SynDEx [6,7]. Both PREESM and SynDEx are based on the Algorithm-Architecture Matching (AAM) methodology. Yet, unlike PREESM, SynDEx is not open source. SynDEx is based on a very specific dataflow model, doesn't support schedulability analysis [27] and its mapping scheduling algorithm is too complex to deal with real use cases. ORCC is an open source tool based on a single dataflow language which is RVC-CAL [28]. PREESM and ORCC do not have the same MoC for the algorithm description. In fact, PREESM uses the predictable PiSDF MoC [29] whereas, the ORCC language is not unpredictable providing results that may differ between two runs. In consequence, the code generated by ORCC has no guarantee in terms of deadlocks and memory requirements.
In an attempt to implement the concept of MTS using PREESM, three items must be developed: the algorithm graph, the architecture graph and the scenario. These items are developed using a graphical interface. In fact, this is the same approach used in Simulink for Matlab or in Labview from National Instruments. Another method that could be used to create these inputs is to use the Higher order Coordination Language (HoCL) [30]. The execution is afterwards based on the tasks used in the workflow graph. This section introduces a complete and detailed description of each element.
Two types of algorithm graphs are supported in PREESM which are PiSDF and IBSDF. They are used to model respectively dynamic and static applications. In fact IBSDF is a static specialization of PiSDF. Both algorithm graphs aim at describing the concept of application with the help of actors and First In First Out (FIFO)s. The FIFOs are used to ensure the communications between the actors. FIFOs control the firing rules and the production/consumption rate between the actors. Each type of algorithm graph has its pros and cons. For example, the IBSDF graph improves the predictability of the dataflow model. In addition, it allows the use of some powerful optimizations at compile-time like the memory footprint reduction and the automatic pipelining. On the downside, the IBSDF graph limits the number of application behaviors that can be modeled. For example, a static graph is not suitable for modeling an algorithm where the execution of certain actors can be turned on or off at runtime, depending on the value of a data token provided by a sensor. The PiSDF model enables this kind of dynamic behaviour giving the possibility of changing the parameter's values at runtime thus, allowing the application to be reconfigurable. In fact, the PiSDF semantics provides a range of parameters and parameter dependencies that allows the change of the consumption and production rates of actors [31] at runtime. Figure 1 shows the different semantics for IBSDF and PiSDF models.
To run the IBSDF model, the only required tool is PREESM; whereas, to run the PiSDF model, an additional tool called Synchronous Parameterized and Interfaced Dataflow Embedded Runtime (SPIDER) must be used. SPIDER is a dataflow based runtime used to support the execution of dynamic PiSDF applications. SPIDER is based on a master/ slave structure in order to facilitate its porting on different multicore architectures [33].
The kind of architecture graph adopted in PREESM is called System-Level Architecture Model (S-LAM). It is a combination of cores related to a shared memory in order to communicate.
The scenario presents the control unit since it contains all the parameters under which the application will perform. The multicore execution is optimized in two steps. In the first one, the scheduling algorithm considers by default that the execution times of all actors are identical. These inaccurate values usually result in an inefficient execution. Therefore, the generated code automatically integrates accurate timing measurements the user report in the scenario for the second step. In the second step, the accurate timing of each actor is used to optimize the multicore mapping scheduling and the generared code.
The workflow is a unit that gathers several tasks. Some of these tasks are responsible of generating the multicore optimized algorithm and others to perform optimizations such as memory optimizations, and automatic pipelining. The automatic pipelining task serves at introducing delays in the critical path in order to increase the throughput of the application on the multicore architecture. [34] details the optimizations provided by the automatic pipelining tool. The memory optimization task provided by PREESM is called the memory footprint reduction. The authors proved in [35] the efficiency of the memory footprint reduction task on a computer vision application, allocating 34% less memory than the state-of-the-art approaches.

IBSDF Models and Granularity Impact
In order to study the efficiency of PREESM tool to explore the parallelism of the MTS concept, two models with different granularity have been developed. The granularity is an important parameter in dataflow models as it affects the performance of the execution. The first IBSDF model shown in Fig. 2 characterized with fine grain granularity and the second one shown in Fig. 3 with a coarse grain granularity. Each model is created based on different levels of parallelism. A first level exposes the parallelism between the MTS concept tasks, and a second level exposes the exploration of the independency between the blocks to be transformed. This section deals with the result given by each model.

Fine Grain Granularity IBSDF Model
The fine grain model (Fig. 2) is composed of two actors with different structure.
The Blocks_to_be_transformed actor is a low level actor. On the other hand, the MTS_concept actor is a hierarchical actor. It encompasses multiple actors internally as illustrated in Fig. 4. Thus, the computational process is divided into several actors. As a result, the execution time of each actor is minimized so that a fine grain granularity model. The fine grain IBSDF model relies on exploring parallelism between MTS concept tasks (DCT-II, DST-VII, DCT-VIII ). Running the application on an x86 machine that has 4 cores yielded a lower execution time on a single core than on 2, 3 and 4 as shown in Table 1. This result introduces a contradiction. In fact, the exploration of parallelism aims at reducing the execution time while increasing the number of cores used. There are several possible reasons for this result. In fact the computing time of the actors is too low compared to the time taken by the tasks of the operating system. In fact, for a fine grain IBSDF model, every process that the operating system runs, including thread creation, communication between tasks, synchronization and memory allocation, affects the performance of the application. Indeed, synchronizations in this model becomes very expensive. For example, PREESM calls barriers at the end of each iteration. Each call takes between 400 nanoseconds, which is not insubstantial at the scale of the actors used for this model. Therefore, in order to ensure the efficiency of the model, the computing time of the actors must be greater than the time required for data transfer and synchronization. In conclusion, the fine grain granularity model is not efficient while using x86 architecture but it could be very useful for multicore embedded systems (FPGA, Kalray,...). Actually, according to the study presented in [36] the fine-grained model has proven its efficiency using Kalray MPPA architecture. In fact, at the intra-cluster level, a fine granularity model is necessary because a large number of tasks must be executed in parallel. Similarly, fine granularity is needed on the FPGA to feed its large number of cores as it explodes the parallelism and gives different levels of parallelism. The fine grain model could leads to a better performance if the MTS concept is applied on multiple blocks at the same time, which is a realistic case as a frame contains a huge number of blocks to be transformed and those blocks are independent. Figure 5 illustrates the results of applying the MTS concept while increasing the number of blocks to be transformed, the number of used cores as well as, applying memory footprint reduction and automatic pipelining optimizations. For memory footprint reduction, a memory script has been added to the transpose actors as their input and output buffers can be allocated in a common memory space. The broadcast actors (the duplicate actors) are automatically associated with memory scripts. The two optimizations boost the speed-up of the application. While increasing the  number of blocks to be transformed the speed-up of the application is getting closer and closer to the theoretical case so that the possibility of creating an efficient model for the MTS concept while employing a coarse grain granularity. The theoretical result is plotted according to Amdahl's law for an application that could be parallelized at 100%; this is why the theoretical result is linear y = x.

Coarse Grain Granularity IBSDF Model
Compared to the fine grain IBSDF model, the coarse grain model is constructed with two non-hierarchical actors in order to ensure that the computing time of the actors is greater than the time required for synchronization and data transfer. The coarse grain model is based on exploring the independency between the blocks to be transformed. In other words, several blocks to be transformed are sent at the same time and the MTS module will be applied to all the blocks simultaneously. For the purpose of studying the efficiency of this second IBSDF model, several tests have been carried out. The tests were performed on x86 architecture which contains 4 cores and another one contains 24 cores. For both tests, the experimental results are close to the theoretical performance (See Figs. 6 and 7). Thus, proving the efficiency of the coarse grain IBSDF model on the x86 architecture. In nutshell, the dataflow modeling of the MTS concept turns to be very beneficial while using a coarse grain model on x86 architecture.
To prove the efficiency of the coarse-grained dataflow model over the state-of-the-art approach used to explore parallelism among the system cores. A comparison between the OpenMP approach and the coarse-grained IBSDF model was performed. OpenMP is commonly used, in shared memory systems, such as the architecture used in the presented model. Therefore, it is interesting to carry out such a comparison. OpenMP often offers specific features, such as preprocessing directives or specific instructions via intrinsics, to target specific architectures. When developing Figure 4 The inside of the MTS_concept actor. with OpenMP, it is necessary to adapt #PRAGMA to the architecture. Often, this is empirically done, by testing the impact of the #PRAGMA parameters on the execution time of the application, which is long and not easy to do. Thus, programming with OpenMP requires a deep understanding of the application, the hardware and the runtime libraries, which takes months to master [37]. However, the coarse grain IBSDF model needs less development time. Also, once the algorithm graph of the application is done, it is easy to switch from one architecture to another without rewriting the code. Figure 8 presents a comparison between the result of running a multicore algorithm created using OpenMP and a multicore algorithm-generated automatically while using PREESM. The results show that the speed-up obtained using the algorithm automatically generated by PREESM is superior to that based on the OpenMP approach. Thus, compared to OpenMP, PREESM builds more efficient model with less development time.

MTS PiSDF Model
In order to implement the multicore optimized MTS concept based on a coarse grain dataflow model in a real encoder or decoder, some modifications must first be made. In fact, in the dataflow models detailed in Sect. 3, the parameters that control the transform order (4,8,16,32,64), and the size of the blocks to be transformed ( width × height ), are modified manually. These parameters cannot be changed at runtime as the presented IBSDF model is a static model. As already detailed in Sect. 2.2 the static property enhances the predictability of the dataflow model and gives the way to perform powerful application optimizations at compile-time, such as memory optimization. While the static property is an important benefit for compile-time optimizations, this property limits the amount of application behavior that can be modeled. On the encoder and decoder sides, there are several blocks to be transformed with different sizes and different transformation orders. In addition, the data dependencies change over time. Thus, the behavior of the application to be implemented must be dynamic and not static. The coarse grain IBSDF model can be implemented in a real encoder/ decoder, but for each combination of block size and transform order, an executable must be created. Then, these executables are controlled via another large one, which is not very convenient. However, by using a PiSDF model, only one executable is needed for all possible combinations.
To create an efficient multicore algorithm for the MTS concept based on PiSDF model is highly required to ensure the efficient dispatching of tasks according to the available processing elements while considering dynamic behavior. For this reason, this section aims to provide a dynamic model of the MTS concept. Also, studying the efficiency of this new model. SPIDER runtime tool has been used to support the execution of this model. When using SPIDER, the application developer only has to provide a dataflow graph and SPIDER takes care of the rest since it is based on pthread. In fact, the developer's work is limited to high-level analysis and manipulation of the application graph (choice of hierarchy level ie. granularity, production/consumption rate, and adding delays between actors). Another powerful aspect of SPIDER is that its optimization leads to high performances. For example, in [38] SPIDER has proved to be able to reduces the execution latency by up to 26% and allows handling multiple executions.
The new model illustrated in Fig. 9 has an additional actor compared to the two IBSDF graphs already detailed. The additional actor named parameters_control as its name indicates, controls the value of the parameters and makes possible their change at runtime, so that a dynamic reconfiguration. The reconfigurable dataflow model promotes a unique trade-off between application dynamicity and predictability, which can be leveraged by a runtime manager (SPIDER) to check application properties or to perform runtime optimizations, such as actor computation mapping [38]. Reconfigurable dataflow models allow actor firing rules to be reconfigured in a non-deterministic way at specific times during the application's execution [39]. The reconfiguration in the PiSDF model is based on parameters. Following the PiSDF execution rules [40], an actor can invoke a reconfiguration of the graph topology and intrinsic parallelism by defining a new parameter value at runtime.
The parameters Transform_order and Block_size affect the performance of the application both during compile-time and runtime. Defining a new value for a parameter in a graph at runtime has a strong impact on the way the actors in that graph will be executed. For example, increasing the value of a parameter used as a production rate by an actor will have an impact on the amount of memory allocated for the output buffer of this actor. By dynamically changing the consumption and production rates of the actors, the number of executions of these actors can be modified, which requires a new mapping of these executions to the processing elements of the architecture. For those reasons an investigation on the impact of the presented PiSDF graph must be performed in order to ensure whether the dynamic coarse grain model still performing on x86 architecture. The parameters_control actor is realized in a manner that provides all possible combinations between the size of the block to be transformed and the order of transformation to be applied. Adding this process to the dataflow model facilitates the implementation of the optimized multicore MTS algorithm in an encoder/decoder in the future. Figure 11 illustrates the result of performing the MTS concept on 80 blocks of different sizes. For all block sizes the execution time on 2, 3, 4 cores are lower than on one core. This result proves the efficiency of the PiSDF model on x86 architecture. Figure 10 illustrates the SPIDER overhead time compared to the execution time of the application according to the number of used cores on an x86 architecture with 4 and 24 cores, respectively. Results show that the SPIDER overhead is too small compared to the execution time for all block sizes. It represents only 0.05% of the entire process. This fact proves the efficiency of the coarse grain model on the x86 architecture and confirms the fact that using a coarse grain model on x86 architecture guarantees that the computation time of the application is higher than the time taken by the operating system. The PiSDF model was also tested on an x86 architecture with 24 cores and with the use of 480 blocks. As shown in Fig. 12 the experimental results for all block sizes were close to the theoretical result (blue curve). In fact, by increasing the number of cores used, the speed-up of the application increases, which corresponds to the objectives of the proposed PiSDF model.

Conclusion
This paper introduced an in-depth case study about the efficiency of applying dataflow modeling on the MTS concept using PREESM. The study was based on two types of algorithm graphs, a static graph IBSDF and a dynamic graph PiSDF. The execution of the PiSDF model requires the use of the SPIDER tool. For both models (IBSDF and PiSDF), the use of dataflow modeling has proven to be effective on the x86 architecture while using a coarse grain model. However, compared to the IBSDF model, the PiSDF model can be easily implemented in a real encoder or decoder. In fact, PiSDF model can handle different combinations of block sizes and transformation orders using the same generated executable, which is not the case for the IBSDF model. The IBSDF model requires the generation of a new executable for each combination. While running the PiSDF model, the result showed a negligible SPIDER overhead time on both architectures (x86 architecture with 4 cores and with 24 cores). It represents 0.05% of the complete process. This fact proves the efficiency of the PiSDF model on both cases and also proves the possibility to switch from one architecture to another without modifying the model and without rewriting any code, unlike classical approaches like OpenMP or OpenCL. The modeling of the MTS concept revealed two important points. The first one is the importance of choosing the level of granularity according to the architecture used, and the second point is to show the difference between IBSDF and PiSDF graphs and their advantages. In future work, we aim to create a PiSDF model for a complete decoder. Also, to use the dataflow models to generate optimized solutions for heterogeneous and embedded architectures (big little Qualcomm SXR2130P processor, Kalray, ...).


VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse

With the recent progress of simulations by 3D modeling software and game engines, many researchers have focused on Embodied AI tasks in the virtual environment. However, the research community lacks a platform that can easily serve both indoor scene synthesis and model benchmarking with various algorithms. Meanwhile, computer graphics-related tasks need a toolkit for implementing advanced synthesizing techniques. To facilitate the study of indoor scene building methods and their potential robotics applications, we introduce INDOORKIT: a built-in toolkit for NVIDIA OMNIVERSE that provides flexible pipelines for indoor scene building, scene randomizing, and animation controls. Besides, combining Python coding in the animation software INDOORKIT assists researchers in creating real-time training and controlling avatars and robotics. The source code for this toolkit is available at https://github.com/realvcla/VRKitchen2.0-Tutorial, and the tutorial along with the toolkit is available at https://vrkitchen20-tutorial.readthedocs.io/en/

Introduction
Simulation engines are tools used by designers to code and plan out a simulation environment quickly and easily without building one from the ground up. As the development of those simulation engines, researchers and game designers are deploying the recent advances in the field of artificial intelligence (AI) to train autonomous and intelligent agents which grow from experimental laboratories into executable products [1].
However, even though learning-based algorithms have gradually increased their influence on training agents in virtual environments, there is still a lack of the toolkit that connects the simulation environment and the state-of-the-art developments in the AI community, including innovative tasks, comprehensive datasets, and powerful algorithms [2]. * These authors contributed equally to this work arXiv:2206.11887v1 [cs.CG] 23 Jun 2022 Figure 1: Toolkit overview. We present this tutorial for our new toolkit INDOORKIT. (1) INDOORKIT supports a wide range of indoor scene datasets synthesized or designed manually, and it allows users to set up their custom dataset. (2) We provide the data processing modules to store scenes that may be from various types of formats into the USD format, making them transferable for other 3D software of game engines. (3) INDOORKIT can be connected with machine learning models for downstream tasks. (Tasks including but not limited to character animation, physical simulation, and robotics.
Recently, with the recent release of NVIDIA OMNIVERSE 1 , a scalable development platform for simulation and design collaboration, researchers can deploy recent advances in AI in OMNIVERSE due to its indispensable features: • Python has taken a lead in determining the programming language for AI or neural networks, and OMNIVERSE exposes much of its functionality through Python bindings; • In OMNIVERSE both physics simulation and the neural network policy training reside on GPU and communicate directly [3]; • The universal scene description (USD) format in OMNIVERSE contains many details in 3D computer graphics scene elements and is supported by a wide range of 3D modeling software (e.g. Blender and Autodesk Maya) and game engines (e.g. Unreal Engine and Unity).
Besides, INDOORKIT is an early work related to OMNIVERSE and we hope that this work paves the way for the creation of influential representation learning in the future. We also provide extensive documentation including a massive amount of demos and tutorials to encourage related research.

Library overview
INDOORKIT is developed by the Center for Vision, Cognition, Learning, and Autonomy at the University of California, Los Angeles. It enables easy data loading and experiment sharing for synthesizing scenes and animation with the Python API in OMNIVERSE. This section briefly describes several features of our INDOORKIT.

Working with 3D scene assets
We integrate multiple indoor scene datasets in INDOORKIT. For datasets with different parameterizations of the body, we include documents for meta-data descriptions and visualization tools to illustrate the characteristics of each dataset. Datasets covered include, but not limited to, 3D-Front [4], scenes of AI2Thor [5], and other indoor scene building pipelines [8]. Besides, to facilitate a contribution to the community, INDOORKIT also provides detailed instructions for users to upload and pre-process their custom datasets.

Why we choose Omniverse
There are several reasons for us to choose the newly platform: OMNIVERSE.
First, for its powerful simulation support: rigid body, soft body, articulated body, and fluid are the main types of simulation that are supported in OMNIVERSE.
• Rigid body: This type simulates the physics in a static environment. Users can set your mass and gravity for each object and assign forces to objects such as friction or buoyancy.
• Soft body: based on the concept of Soft Body Dynamics (SBD), the soft body is controlled by mathematical equations which describe how physical objects behave in the real world when they collide with other objects or themselves. The problem with traditional computer graphics is that the physics simulation can only be done by hand and it takes a lot of time to get an accurate result. With OMNIVERSE: soft body this task can be done automatically.
• Articulation body: working with OMNIVERSE, INDOORKIT provides the tool that enables us to build physics articulations such as robotic arms, kinematic chains, and avatars that are hierarchically organized. It also helps us get realistic physics behaviors in the context of simulation for industrial applications.
• Fluid: OMNIVERSE allows you to simulate the behavior of liquids and gases. It's based on the Navier-Stokes equations, which describe how fluids flow in an environment. INDOORKIT connects the set up of fluid with indoor scene assets, which provides better flexibility of custom tasks.
Second, for its Python scripting environment, it is easy to bring open-source and third-party Python libraries into OMNIVERSE to help the research.
• Python is a general-purpose programming language that can be used for many different tasks. It has a simple syntax and readable code, and it's easy to learn. Python is also very popular in the industry, because of its high performance, robustness, and versatility.
• Deep learning can be applied to many different fields from computer vision, speech recognition, natural language processing (NLP), translation, robotics, and much more. Deep learning with Python using Pytorch [10] and Tensorflow [11] libraries can be easily supported in OMNIVERSE.
Third, OMNIVERSE has the powerful rendering ability with the ray tracing technology.
• Ray tracing is a system that improves the lighting in the simulation. It cab be used in everything from reflections to shadows both in the environment and scene elements including atmospheric effects, surfaces reflections and even diffused lighting.
• Besides, users can render the scenes with the universal scene description (USD) format. This is a highly compressed format that allows to store the entire scene information in a single file. It is also very simple to use and understand. The basic idea behind USD is that it stores all of the information about how the scene is rendered, including models, textures, lights, cameras, and animation.

Potential use case
Another essential goal of INDOORKIT is to apply the state-of-the-art embodied AI research while synthesizing photo-realistic and physical-realistic rendering. Here, we list a few common use cases of our library. The full demo and tutorial for them can be found at https://github.com/realvcla/VRKitchen2.0-Tutorial

Indoor scene labeling
Indoor scene labeling is the process of identifying and labelling indoor scenes. The goal of this project is to improve the quality, quantity and consistency of indoor scene labels in an effort to facilitate more efficient research use. The field of the EAI still lacks of meaningful and near-realistic indoor scenes. We apply OMNIVERSE platform to perform labeling steps for indoor scenes. The sampled data piece contains the information of the game, and allows users to perform downstream tasks. INDOORKIT also offers a clean interface for labeling scene and robot information.

Indoor character animation
With the recent progress in 3D character animation creation, the popularity of animation generation, as well as its application, grows. Meanwhile, with the improvement of virtual environment simulation capability, researchers have started to study agent behavior in the context of Embodied AI. However, the production process of adapting the generated animation in a photo-realistic and physics-reliable environment is laborious. The key idea is to make a balance between the original animation clip, physics scene, and social interaction meaning through reinforcement learning on the kinematics.

Robotics oriented simulation
Robotics is a field of study that involves the design, construction and operation of robots. The goal of robotics in OMNIVERSE is to build machines capable of performing tasks that are difficult or impossible for humans to perform. Robotics can be used in many different fields including medicine, manufacturing, research and education.

Development and maintenance
The project is developed by the Center for Vision, Cognition, Learning, and Autonomy at University of California, Los Angeles. INDOORKIT is developed publicly through Github with an issue tracker to report bugs and ask questions. Documentation consists of tutorials, examples, and API documentation. The third-party packages include Pytorch [12] for the deep learning framework, and Jupyter notebook [13]. For the demo and tutorial, please visit https:// vrkitchen20-tutorial.readthedocs.io/en/latest/..

Conclusion
We presented INDOORKIT, an Python library to help researchers to easily develop their indoor scene synthesizing methods and apply the scene to EAI tasks.