Emotion recognition based on multi-modal physiological signals and transfer learning

In emotion recognition based on physiological signals, collecting enough labeled data of a single subject for training is time-consuming and expensive. The physiological signals’ individual differences and the inherent noise will significantly affect emotion recognition accuracy. To overcome the difference in subject physiological signals, we propose a joint probability domain adaptation with the bi-projection matrix algorithm (JPDA-BPM). The bi-projection matrix method fully considers the source and target domain’s different feature distributions. It can better project the source and target domains into the feature space, thereby increasing the algorithm’s performance. We propose a substructure-based joint probability domain adaptation algorithm (SSJPDA) to overcome physiological signals’ noise effect. This method can avoid the shortcomings that the domain level matching is too rough and the sample level matching is susceptible to noise. In order to verify the effectiveness of the proposed transfer learning algorithm in emotion recognition based on physiological signals, we verified it on the database for emotion analysis using physiological signals (DEAP dataset). The experimental results show that the average recognition accuracy of the proposed SSJPDA-BPM algorithm in the multimodal fusion physiological data from the DEAP dataset is 63.6 and 64.4% in valence and arousal, respectively. Compared with joint probability domain adaptation (JPDA), the performance of valence and arousal recognition accuracy increased by 17.6 and 13.4%, respectively.

In emotion recognition based on physiological signals, collecting enough labeled data of a single subject for training is time-consuming and expensive. The physiological signals' individual differences and the inherent noise will significantly affect emotion recognition accuracy. To overcome the difference in subject physiological signals, we propose a joint probability domain adaptation with the bi-projection matrix algorithm (JPDA-BPM). The bi-projection matrix method fully considers the source and target domain's different feature distributions. It can better project the source and target domains into the feature space, thereby increasing the algorithm's performance. We propose a substructure-based joint probability domain adaptation algorithm (SSJPDA) to overcome physiological signals' noise effect. This method can avoid the shortcomings that the domain level matching is too rough and the sample level matching is susceptible to noise. In order to verify the effectiveness of the proposed transfer learning algorithm in emotion recognition based on physiological signals, we verified it on the database for emotion analysis using physiological signals (DEAP dataset). The experimental results show that the average recognition accuracy of the proposed SSJPDA-BPM algorithm in the multimodal fusion physiological data from the DEAP dataset is 63.6 and 64.4% in valence and arousal, respectively. Compared with joint probability domain adaptation (JPDA), the performance of valence and arousal recognition accuracy increased by 17.6 and 13.4%, respectively.

Introduction
Emotion is a complex expression that integrates people's psychological and physiological functions. It reflects the subjective response of individuals to external stimuli all the time (Sharot et al., 2004). Since affective computing was proposed, researchers have devoted to digitizing the concept of emotion, enabling computers to recognize and process it, and providing more reliable signal input for human-computer interaction (Picard, 2003;Mühl et al., 2014). In the humancomputer interaction system, accurately decoding the user's emotion can make the device not only passively receive the user's instructions but also truly perceive the user's state, to better understand the user's intention and establish a more natural and harmonious human-computer interaction environment (Egger et al., 2019). As a research hotspot in human-computer interaction, affective computing is widely used in traffic safety Du et al., 2020), brain-computer interface (Al-Nafjan et al., 2017;Rao et al., 2018), medical health (Hosseinifard et al., 2013;Huang et al., 2019), and other fields. Affective computing includes three continuous processes: emotion recognition, behavior generation, and induction. Accurate emotion recognition is the basis for building a good human-computer interaction experience (Egger et al., 2019). However, in practical applications, collecting large numbers of data for each user to train the classifier is difficult, and the recognition accuracy is easily affected by data noise (Wan et al., 2021). When the accuracy of emotion recognition is influenced by physiological signals' individual differences and inherent noise, making the model trained in the existing data set accurately identify new users' emotions without collecting data or collecting as little data as possible has essential research value and application significance.
Nowadays, there are many emotion recognition methods, such as analyzing users' voices Shaqra et al., 2019), facial expressions (Lawrence et al., 2015;Abdulsalam et al., 2019), and physiological signals (He et al., 2017;Liao et al., 2020). Physiological signals are the most easily acquired signals by the human body through sensors. It contains many important physiological and psychological information about the human body and plays a significant role in computer recognition of human emotions . Compared with emotion recognition based on facial expression, emotion recognition based on physiological signals not only has the advantages of low cost and high efficiency in data acquisition but also can avoid the errors caused by light and shadow acquisition and the invasion of user privacy (Hao et al., 2020;Fu et al., 2021).
In the aspect of emotion recognition, electroencephalogram (EEG) has been paid more attention by researchers among many physiological signals. The analysis of EEG signals in the field of emotion recognition depends on data preprocessing, feature extraction, and feature classification (Xie et al., 2021). Many researchers use traditional machine learning or deep neural network to classify EEG signals by extracting the energy features of the delta, theta, alpha, beta, and gamma bands. For example, Verma and Tiwary (2014) extracted the relative power energy, logarithmic relative power energy, absolute logarithmic relative power energy, standard deviation, and spectral entropy features of five frequency bands from EEG signals. Liu et al. (2016) used a deep autoencoder to extract the features of EEG signals in the DEAP dataset and extract features. Sorkhabi (2014) used continuous wavelet transform to extract energy features of five frequency bands and entropy features of wavelet coefficients. Yin et al. (2017) extracted the frequency band power features, statistical features, signal zero crossing rate, Shannon entropy, spectral entropy, kurtosis, skewness, and other features of the five frequency bands. Torres-Valencia et al. (2017) extracted statistical features of EEG signals and power features of five frequency bands.
However, a single EEG signal's lack of feature information will lead to low emotion recognition accuracy. Some researchers use feature level fusion or signal level fusion to fuse multimodal signals to improve emotion recognition accuracy. He et al. (2021) extracted 11 features from the EEG signal of FP2 channel and 6 features from HR. Using multi-core learning for fusion, they achieved 67% binary classification accuracy under fewer signals and channels. Song et al. (2019) used attention-based long-term short-term memory to fuse multimodal physiological signals, including electroencephalogram (EEG), Galvanic Skin Response (GSR), respiration (RSP), and electrocardiogram (ECG), to improve the classification accuracy. Our study uses EEG, RSP, GSR, and photo-plethysmograph (PPG) signals collected from the database for emotion analysis using physiological signals (DEAP) (Koelstra et al., 2011) for feature extraction. We concatenate four mode features to achieve multimodal feature fusion. It can remedy the inherent limitations of the single mode by providing more dimensional features. Consequently, the multimodal features improve the accuracy of emotion recognition.
The EEG signals are very complex due to the inherent nonstationary, non-linear, and non-Gaussian characteristics (Subha et al., 2010). Meanwhile, EEG signals are greatly affected by age, psychology, and other factors, which result in significant differences in individual EEG signals (Lotte et al., 2018). This difference is often substantial and cannot be ignored. The traditional emotion recognition based on EEG does not consider the existence of differences and directly trains a general model. The difference between EEG signals of different individuals will directly affect the accuracy of model recognition and classification and lead to a poor generalization ability of the model . Considering different types of information in EEG signals make it difficult to filter out information sensitive to specific tasks, and there are few similar EEG data among different individuals due to the significant difference in EEG, it is problematic that use the deep learning Frontiers in Neuroscience 02 frontiersin.org model based on old user data training to estimate the mental state of new users (Wan et al., 2021). In order to solve the above problems, some researchers have introduced transfer learning to emotion recognition.  proposed a multi-source transfer learning algorithm to transfer the existing emotion model to new subjects. The experimental results show that this method can effectively reduce the demand for data quantity and increase the calibration capability of the model. Chai et al. (2017) proposed an adaptive subspace feature matching algorithm for emotion recognition, which aligns the source and target subspaces by learning linear transformation to reduce the distribution discrepancy between the source and target domains. Lin and Jung (2017) proposed a conditional transfer learning framework. The algorithm first evaluates the individual's transferability to positive transfer and then selectively leverages the data from others with comparable feature spaces. Therefore, in order to solve the low accuracy of emotion recognition caused by the mismatch between individual specificity and global threshold, we introduce domain adaptation, a transfer learning method, into emotion recognition. This method can apply the patterns learned in one domain to other domains and reduce the differences in EEG data distribution so that to improve the model's ability to recognize new users' emotions.
Generally, the domain adaptation method usually seeks the alignment between the source and target domains. Different domain adaptation methods often use different alignments. The current alignment methods can be divided into three categories according to distribution matching schemes: domain-level, class-level, and sample-level . Pan et al. (2010) proposed the transfer component analysis (TCA) method, which uses the maximum mean difference (MMD) to learn a transformation matrix in the reproducing kernel Hilbert space (RKHS) to align the marginal distribution between the two domains. Long et al. (2017) proposed the joint distribution adaptation (JDA) method to align the joint distribution of multiple domains through multi-kernel MMD. Sun et al. (2017) proposed the correlation alignment (CORAL) method, which minimizes the domain shift by aligning the second-order statistical data of source and target distribution. The above commonly used domain adaptation methods belong to domainlevel matching. The domain-level matching completely ignores the intra-domain data structure. It is too rough to miss some details and challenging to achieve good matching results.
The sample level matching can avoid the problem that domain-level matching ignores intra-domain data structure. Courty et al. (2017) proposed a regularized unsupervised optimal transport model, which uses the optimal transport theory to calculate the distance between the probability distributions of the source and the target domain. In the research of Das and Lee (2018), the source and the target domain are regarded as hypergraphs, and the first-order, second-order, and third-order similarities between graphs are used for class-regularized hypergraph matching to obtain the matching between the samples of the source domain and the target domain. However, sample level matching is very timeconsuming, and it is more prone to overfitting when local information is affected by the noise.
Class-level matching can neutralize too rough domainlevel matching and too fine sample-level matching. Wang et al. (2018) proposed the Stratified Transfer Learning (STL) method. STL transforms the same classes in the source and the target domain into the same subspace and uses the intraaffinity of the class to perform knowledge migration within the class. Tian et al. (2020) proposed the Centroid Matching and local Manifold Self-learning (CMMS) method. CMMS can thoroughly explore the data distribution structure of the domain and minimize the distribution difference in domain adaptation by combining class centroid matching with local manifold self-learning. Lu et al. (2021) proposed a domain adaptation method based on substructure level matching, which regards a class as synthesizing multiple substructures and aligning the substructures. The above commonly used domain adaptation methods belong to class-level matching. Considering that the EEG signal acquisition process contains the location information of different channels, which has the intra-domain data structure, we adopt the class-level domain adaptation to avoid rough alignment of domain-level adaptation and overfitting of sample-level adaptation.
In the matching process of the source and target domains, it is necessary to project the source and target domains into the same feature space through the projection matrix. The TCA, JDA, BDA, and JPDA all uses the single projection matrix for transfer (Pan et al., 2010;Long et al., 2017;Wang et al., 2017;. However, the distribution of the source domain and target domain is different, and a single projection matrix cannot account for all the feature distribution of the source and target domains. Therefore, we propose a biprojection matrix (BPM) to better project the source and target domains into the feature space.
This paper uses EEG, RES, PPG, and GSR signals collected from the DEAP dataset to extract features, and we concatenate four mode features to achieve multimodal feature fusion. Multimodal fusion gives full play to the advantages of each mode and makes up for its inherent limitations, improving the accuracy of emotion recognition. In order to improve the generalization ability of the model, we propose a joint probability domain adaptation method based on the substructure. Substructure-level data is aligned by discriminative joint probability maximum mean discrepancy (DJP-MMD) . Substructure-based joint probability domain adaptation (SSJPDA) can avoid inadaptability caused by rough matching and overfitting when learning local information caused by noise points. In order to better project the source and target domains, we propose a method of the bi-projection matrix (BMP), which can effectively avoid data loss in the projection stage.
The main contributions of this study are as follows: We proposed a substructure-based joint probability domain adaptation method (SSJPDA). We proposed the bi-projection matrix (BPM) method and applied it to the SSJPDA algorithm. We validated the SSJPDA algorithm and the SSJPDA-BPM method on DEAP dataset.
The rest of this paper is arranged as follows: Section "Materials and methods" introduces the SSJPDA with the BPM algorithm. Section "Results" presents the results verified on the DEAP dataset. Section "Discussion" gives the full discussion above the result.

Physiological signal dataset
This study adopted the DEAP dataset to inspect our proposed algorithm. DEAP dataset was established by Koelstra et al. (2011) in 2012 and contained 32 subjects. Every subject watched the 40 selected music videos, and each video viewed by the subjects was regarded as an independent experiment. After the video viewing, the subjects need to use the self-evaluation model to score arousal, valence, like/dislike, dominance, and familiarity, providing label information for each signal. Every experiment recorded 40 physiological signals of subjects, of which the first 32 signals were EEG signals collected according to the international 10-20 system, and the remaining 8 signals were peripheral physiological signals, including 2 ophthalmic signals, 1 skin electrical signal, 2 EMG signals, 1 respiratory record, 1 plethysmography, and 1 temperature record. The dataset also preprocessed the collected signals. Each test section's EEG data and other peripheral physiological signal data were divided into 3 s baseline data and 60 s test data. EEG signals are collected according to the international 10-20 lead system and downsampling from 512 Hz original sampling frequency to 128 Hz. RES, PPG, and GSR signals are down-sampled to 128 Hz. A band-pass frequency filter of 4-45 Hz and a blind source separation technique were used to remove the eye artifacts.

Feature extraction
Considering that the subjects are not always in a high emotional activation state if the sliding window is used to divide the data into small segments, many segments will contain useless information (Piho and Tjahjadi, 2018). Therefore, we directly extract features from the preprocessed 60 s experimental data to make samples instead of dividing continuous data into multiple segments and making each segment into samples in the feature processing. We extract the differential entropy features of five frequency bands from each recorded EEG data from each EEG channel. These five frequency bands are related to people's state of mind, so they also contain information about the state of specific thinking tasks. These five bands are Delta (1-4 Hz), Theta (4-8 Hz), Alpha (8-13 Hz), Beta (13-30 Hz), and Gamma (30-48 Hz). Some studies have shown that the differential entropy feature is superior to the power spectral density feature (PSD) in EEG-based emotion recognition (Zheng and Lu, 2015;Soleymani et al., 2017).
We extract their time-domain and frequency-domain features for the peripheral physiological signals PPG, GSR, and RES. The extracted time-domain features and frequencydomain features refer to numerous previous studies (Verma and Tiwary, 2014;Yin et al., 2017;Zhang et al., 2021). Timedomain features depend on statistical features, which are simple and intuitive. It realizes classification by analyzing statistical features such as mean, maximum, minimum, root mean square, standard deviation, etc. The time-domain analysis contains all the characteristics of physiological signals, and the signal is processed directly. Hence the loss of information is relatively small. For example, from the time domain characteristics of PPG signals, we can analyze the heart rate and its changes, which are closely related to emotional arousal. In addition, Frequency domain features can show the frequency information that timedomain features cannot reach in more detail. Consequently, we got 1,280 samples (32 subjects × 40 samples). Table 1 lists the features extracted from the data.

The generation of substructures
containing the label recorded as D s . The target domain X t = {x t,j } n t j=1 without label recorded asD t . The n s and n t are the number of source domain samples and target samples, respectively. x ∈ R d×1 is the feature vector, and y ∈ {1, . . . , C} is its label in the Cclass classification problem. D s and D t have the same feature space and label space, but the feature distribution is different, i.e., P(X s , Y s ) = P(X t , Y t ). The task of domain adaptation is to reduce the distribution difference between the source domain and the target domain, so as to predict the label y t of the target domain D t with the help of the source domain D s .
We use δ ∼ N 0; σ 2 and X to represent all feature data and the Gaussian mixture model (GMM) to fit them. The kth component in GMM is recorded as X k ∼ N (z k , σ k ) where z k represents mean value and σ k represents covariance. Our goal is to get mean value z k and covariance σ k . These GMM parameters can be obtained using the Expectation Maximum (EM) algorithm. Suppose K s and K t are the number of GMM components in the source domain and the target domain, respectively. K s is determined by the Bayesian Information Criterion (BIC), and K t is manually set according to the specific data set.
After obtaining the GMM of the source domain and the target domain, we regard each component of the GMM as a substructure in the feature space, and the information of the cluster center represents the substructure. Specifically, set where µ s and µ t are the distribution of source domain and target domain, respectively. z ∈ R d×1 is cluster center, and δ z is the Dirac function at location z. wis the probability weight The cost between z i and z j in square Euclidean distance can be expressed as Therefore, the problem can be regarded as the partial optimal transmission (POT) problem, and the upper bound w s,i is 1. The total cost of POT is π, C F that is the Frobenius dot product of cost matrix C and coupling matrix π. The C ∈ R k s ×k t represents the cost of µ s and µ t distribution, and theπ ∈ R k s ×k t represents the coupling between µ s and µ t distribution.
The goal is to obtain the optimal transmission, which can be expressed as where H(π) = ij π ij log π ij is the entropy term, and λ 1 is the super parameter to balance the speed and accuracy calculation. The feasible solution set of π T 1 k s = w t is C 1 , and then it can be solved by the Lagrange method. Thus, we can easily get the optimal π * .
where π 0 = exp − C λ 1 − 1 and represent element-wise divide and diagrepresents the diagonals. Once the coupling matrix π * 1 is obtained, the source domain weights can be easily calculated as w s = π * 1 1 k t .

Substructural joint probability domain adaptation
The domain adaptation (DA) method attempts to find a mapping h. The source domain and target domain are mapped to the same subspace, so that the classifier trained on h(x s )can achieve good classification effect on h(x t ). For example, a linear maph(x) = A T xfor the source and the target domains, where A ∈ R d× p , p ≤ d.
Due to the difference between the source domain and the target domain, it is generally assumed that their probabilities distributions are not equal. The derivation of TCA, JDA and BDA algorithms are based on the inequality of the marginal probabilities P(X s ) = P(X t ) or the conditional probabilities P(Y s |X s ) = P(Y t |X t ). However, the JPDA algorithm derives from the inequality assumption of joint probabilities P(X s , Y s ) = P(X t , Y t ). Because JPDA directly considers the difference of joint probability distribution, the performance of JPDA is better than the traditional DA method, which JPDA can improve the between-domain transferability and the between-class discrimination .
After obtaining the substructure, the set of substructures in source domain is recorded as {Z S , Y S } = {(z s,i , y s,i )} k s i=1 , and the set of substructures in target domain is recorded as Z t = {z t,j } k t j=1 , where k s and k t are the number of source domain substructure and target domain substructure, respectively.
Let the source domain substructure one-hot coding label matrix be Y s = [y s,1 ; . . . ; y s,k s ] and the predicted target domain substructure one-hot coding label matrix beŶ t = Frontiers in Neuroscience 05 frontiersin.org [ŷ t,1 ; . . . ;ŷ t,k t ] where y s,ks ∈ R 1×C andŷ t,kt ∈ R 1×C . Define where Y s (:, c) denotes the c-th column of Y s , Y s (:, c) * (C − 1) repeats Y s (:, C) C−1 times to form a matrix in R k s ×(C−1) , and Y t (:, 1 : C)ˆc =c is formed by the 1st to the C-th, (except the c-th) columns of Y t . Clearly, F s ∈ R k s ×(C(C−1)) andF t ∈ R k t ×(C(C−1)) . F s is fixed, andF t is constructed from the pseudo labels, which are updated iteratively. Therefore, the objective function of JPDA can be written as follows: where µ > 0 is a trade-off parameter and λ is a regularization parameter. N s , N t , M s and M t are defined as where H = I − 1 k is the centering matrix, in which k = k s + k t and 1 k ∈ R k×k is a matrix with all elements being 1 k . Let Z = [Z s , Z t ], then we reach the Lagrange function of Eq. 8 where η is Lagrange multiplier, and R max and R min have dimensionality k × k. By setting the derivative ∇ A J = 0, Eq. 17 becomes a generalized eigen-decomposition problem: A is then formed by the p trailing eigen-vectors. A classifier can then be trained onA T Z s and applied to A T Z t . The pseudocode of SSJPDA for classification is summarized in Algorithm 1.

Input:
X S and X t , source and target domain feature matrices; Y S , source domain one-hot coding label matrix; µ, trade-off parameter; λ, regularization parameter; T, number of iterations; Output: Y t , estimated target domain labels.

Begin:
Use EM for GMM, cluster each class data in the source to obtain ,and cluster the unlabeled data in target domain to obtain Z t = {z t,j } k t j=1 ; Compute cost matrix C and coupling matrix π using Eq. 3 and Eq. 4, respectively; Compute the weights of source substructures w s = π * 1 1 k t and target substructures w t = 1 k t k t for n = 1,..., T do Construct the joint probability matrix R min and R max by Eq. 12 and Eq. 13; Solve the generalized eigen-decomposition problem in Eq. 14 and select the p trailing eigenvectors to construct the projection matrix A; Train a classifier f on (A T Z s , Y S ) and apply it to A T Z t to obtain Y t = {y t,j } k t j=1 which is the label matrix of substructure in target domain Z t = {z t,j } k t j=1 End for For each substructure z t,j , assign its label y t,j to all samples it contains, and getsŶ t = {y t,j } n t j=1 End Algorithm 1. Substructural Joint Probability Distribution Adaptation (SSJPDA) Substructure-based joint probability domain adaptation algorithm with bi-projection matrix As described in the previous subsection, the source and target domains have different probability distributions, so applying only a single projection matrix to both domains simultaneously may lack the ability to align their probability Frontiers in Neuroscience 06 frontiersin.org distributions well. It is better to make the source domain and the target domain have their own projection matrix to accomplish the distribution alignment task together. On this basis, we take SSJPDA algorithm as an example to explain how to design the projection matrix of source domain and target domain, respectively, and call it SSJPDA-BPM. Donate the projection matrices of the source domain and the target domain asA s and A t , respectively. Therefore, the objective function of SSJPDA-BPM can be written as follows: where H s = I k s − 1 k s (or H t = I k t − 1 k t ) is the centering matrix, in which 1 k s ∈ R k s ×k s (or 1 k t ∈ R k t ×k t ) is a matrix with all elements being 1 k s (or 1 k t ).
, then we reach the Lagrange function of Eq. 15 where η s η t are Lagrange multipliers, and By setting the derivative ∇ A s J = 0, ∇ A s J = 0, and add a constraint Z s R 12 Z T t A s =Z t R 21 Z T s A t , then Eq. 16 becomes two generalized eigen-decomposition problem: A s and A t are then formed by the p trailing eigen-vectors of each problem. A classifier can then be trained on A T s Z s and applied to A T t Z t . The pseudocode of SSJPDA-BPM for classification is summarized in Algorithm 2.

Input:
X S and X t , source and target domain feature matrices; Y S , source domain one-hot coding label matrix; µ, trade-off parameter; λ, regularization parameter; T, number of iterations; Output: Y t , estimated target domain labels.

Begin:
Use EM for GMM, cluster each class data in the source to obtain {Z S , Y S } = {(z s,i , y s,i )} k s i=1 ,and cluster the unlabeled data in target domain to obtain Z t = {z t,j } k t j=1 ; Compute cost matrix C and coupling matrix π using Eq. 3 and Eq. 4 respectively; Compute the weights of source substructures w s = π * 1 1 k t and target substructures w t = 1 k t k t for n = 1,..., T do Construct the joint probability matrix R in Eq. 17 Solve the generalized eigen-decomposition problem in Eq. 18 and Eq. 19, and select the p trailing eigenvectors to construct the projection matrix A s and A t ; Train a classifier f on A T s Z s and applied to A T t Z t to obtain Y t = {y t,j } k t j=1 which is the label matrix of substructure in target domain End for For each substructure z t,j , assign its label y t,j to all samples it contains, Validation of the substructure-based joint probability domain adaptation algorithm and substructural joint probability distribution adaptation with bi-projection metrix The DEAP dataset contains 32 subjects, each taking turns as the target domain and the remaining 31 people as the source domain. The number of samples in the source domain is 1,240 (31 subjects × 40 samples), and the number of target domain samples is 40 (1 subject × 40 samples). After dividing the source and target domains, the EEG, GSR, PPG, and RES modes were transferred, respectively, and all the subjects' valence and arousal dimensions were classified, respectively. In each sample, the feature dimension of EEG is 160, the feature dimension of GSR is 7, the feature dimension of PPG is 8, and the feature dimension of RES is 8. Those four modes were fused through average splicing, where the feature dimension after fusion in each sample is 183. The feature dimension of the modes remains the same dimension before and after the transfer learning. The effects of single-mode transfer and multi-mode transfer are compared to explore whether data fusion can promote the accuracy of the transfer learning algorithm. By comparing SSJPDA with other transfer learning methods and traditional machine learning methods, this paper explores whether SSJPDA can improve recognition accuracy. Hyperparameters of the model will affect the recognition accuracy. We divide the target domain with 40 samples from 1 subject into a verification set and a test set for the specific hyperparameter configuration in the algorithm, which follows similar protocols used in Courty et al. (2016). Among them, the training set is an optional 10 samples, and the test set is the remaining 30 samples. Both validation and test sets have no labels. The validation set data and source domain data are trained together to obtain the best accuracy within the range of hyperparameters, and the range of hyperparameter sets follows (Kerdoncuff et al., 2021). Under the best hyperparameters set, the classification accuracy and F1 measure are used to measure the performance of our proposed algorithm on the test set.

Result Experiment 1
In Experiment 1, JPDA, JPDA (BMP), SSJPDA, and SSJPDA (BMP) algorithms were used to transfer EEG, PPG, GSR, RES, and four-mode fusion data (ALL) of subjects, respectively. Table 2 shows the average accuracy and F1-measure of 32 subjects in valence and arousal. Table 2 shows that in the DEAP dataset, the recognition accuracy of multimodal fusion data is less improved than that of single-mode data recognition. Even in the identification of some modes of JPDA and JPDA-BPM, the accuracy of single-mode is higher than that of multi-mode. However, this phenomenon does not appear in the domain adaptation algorithm using substructure. In the classification of valence and arousal by SSJPDA and SSJPDA-BPM algorithms, the recognition accuracy and F1-measure based on multimodal data are generally higher than that of single-mode data. In the recognition of multimodal data, the recognition accuracy of SSJPDA and SSJPDA-BPM in valence is 14.1 and 19.3% higher than that of JPDA and JPDA-BPM, respectively. In the recognition accuracy of arousal, SSJPDA and SSJPDA-BPM are higher than JPDA and JPDA-BPM by 11.8 and 12.4%, respectively. In the single-mode recognition, SSJPDA-BMP has higher recognition accuracy and F1 than JPDA-BMP in every single mode. Similar rules also appear in the comparison between SSJPDA and JPDA. By comparing the recognition ability of the two transfer learning algorithms with or without the BPM algorithm in each mode, we find that the BPM algorithm is more effective in the transfer learning algorithm with substructure. Among the algorithms that do not use substructures, whether to use the BPM algorithm has little impact on transfer performance.
In order to present the representations generated by different methods more intuitively, we use the t-SNE algorithm in multimodal data experiments to reduce the dimension and visualize the representations generated by different algorithms. Figure 1 is the t-SNE diagrams of each algorithm in Experiment 1 on multimodal data. The dots legend represents the source domain data, and the legend of the star represents the target domain data. The light blue and dark blue represent positive samples, and the orange and red represent negative samples.
According to Figure 1, the representations generated by different algorithms have consistent performance, regardless of valence or arousal classification. The substructures generated by SSJPDA and SSJPDA-BPM through clustering in the domain can significantly reduce the quantity of data. JPDA-BPM and SSJPDA-BPM can lessen the intra-class sample distance and increase the inter-class sample distance in the same domain. At the same time, they can make the same kind of samples in different domains align better compared with not using the BPM algorithm. The representation generated by SSJPDA-BPM has better separability than others.

Experiment 2
The source domain data and target domain data settings of Experiment 2 are the same as Experiment 1, but only fusion data is used for comparison in the different algorithms. Traditional machine learning and transfer learning algorithms are used to classify valence and arousal. Because the TCA, JDA, BDA, and JPDA algorithms all use the 1-Nearest Neighbor (1NN) model in classification, we choose 1NN as the traditional machine learning model to compare the impact of the transfer learning algorithm on recognition results. Table 3 shows the average accuracy and F1-measure of 32 subjects using different algorithms in valence and arousal. Table 3 shows that in the problem of emotion recognition based on the DEAP dataset, when the data distribution of the source domain and target domain is different, the performance of all transfer learning algorithms is better than the 1NN algorithm. In recognition of valence and arousal, the algorithm with the worst classification accuracy in the transfer learning algorithm is still 1.2% (TCA) and 2.2% (JDA) higher than 1NN, respectively. We proposed SSJPDA-BPM algorithm has the best performance. The recognition accuracy and F1measure values of valence are 63.3 and 65.3%, respectively. The recognition accuracy and F1-measure arousal values are 64.4 and 67.9%, respectively. Its accuracy and F1-measure values are higher than other algorithms. Compared with the traditional transfer learning algorithm, SSJPDA-BPM has higher classification accuracy than TCA, JDA, and BDA by 29.8, 28.2, and 22.5%, respectively, in valence classification. In the recognition accuracy of arousal, SSJPDA-BPM is 23.6, 25.1, and 19.7% higher than TCA, JDA, and BDA, respectively. The comparison results of whether to use BPM and SS algorithms have been described in detail in Experiment 1, which will not be explained in this part. Figure 2 is the line chart showing the recognition accuracy of each algorithm in Experiment 2 in 32 subjects in descending order, of which Figure 2A is the recognition accuracy of valence and Figure 2B is the recognition accuracy of arousal. The gray horizontal line is the chance level of 50% for the two classes. Each color corresponds to an algorithm. Subjects above the gray level line are represented by upward triangles. The recognition accuracy of this subject in the algorithm is higher than that of the chance level. Downward triangles represent subjects below the gray level line, and the recognition accuracy of this subject in the algorithm is lower than the accuracy of the chance level.  Figure 2A shows that more than half of the subjects have a recognition accuracy higher than the chance level of 50% for two classes in recognition of valence by the 1NN, TCA, and JDA algorithms. The recognition accuracy of 1NN and TCA in some subjects is less than 30%. Therefore, the average recognition accuracy of these two algorithms is lower than JDA. By comparing JDA, BDA, and JPDA algorithms in order of this arrangement, we can see that the number of people whose three algorithms are higher than the chance level of 50% is slowly increasing. Meanwhile, the highest and lowest recognition accuracy of subjects in the test set is also gradually increasing. The performance of JPDA-BPM is lower than that of JPDA. Although JPDA-BPM algorithm has more subjects with recognition accuracy higher than 70 and 60%, wrong matching still leads to more subjects with recognition accuracy lower than 45%. The SSJPDA and SSJPDA-BPM algorithms have improved compared to the original algorithm. It is worth noting that the recognition accuracy of the SSJPDA-BPM algorithm is above 55% in all subjects. Figure 2B shows that the number of subjects with arousal recognition accuracy higher than the chance level exceeded half of the total sample size. 1NN, TCA, and JDA algorithms have more than 70% recognition accuracy in some subjects. However, the recognition performance of the algorithm is poor in some subjects, and its recognition accuracy is lower than 35%, which leads to the low average recognition accuracy of these three algorithms. In the JPDA-BPM algorithm, one subject has a recognition accuracy of 85%, which is the highest among the eight algorithms. Meanwhile, its minimum recognition accuracy is 35%, and the number of people lower than the chance level of 50% is also higher than JPDA, which leads to little difference between its average recognition accuracy and JPDA. In the comparison between SSJPDA and SSJPDA-BPM, the performance of SSJPDA-BPM is generally higher than SSJPDA, and the recognition accuracy is lower than SSJPDA only in a few subjects. Comparing whether to use the SS method, SSJPDA and SSJPDA-BPM have been improved compared to the original algorithm, and the recognition accuracy of all subjects is above 50%.

Discussion
The performance of different algorithms with multi modal and single modal data In the algorithm based on non-substructure, the recognition accuracy using multimodal data is less improved than that using single-mode data. In the substructure-based algorithm, multimodal data can significantly improve recognition performance. Multimodal data in JPDA, the amount of data in the source domain and target domain are very different. The source domain consists of 31 subjects, each of which contains 40 samples. The target domain is 40 samples from one subject, of which 40 samples are also divided into a validation set composed of 10 random samples and a test set consisting of 30 random samples. Therefore, there is an enormous difference in the data volume between the source domain and the target domain. When the source and the target domain are projected to the same feature space, the probability of false matching will increase, which affects transfer recognition's accuracy.
Fusing the features of the four modes will increase the sample dimensions of the source and the target domain. The probability of sample error matching is greater than that of single-mode identification, so the performance of the nonsubstructure algorithm in multimodal data identification is poor. The transfer learning algorithm based on substructure can avoid error matching caused by sample dimensions increasing and data volume differences between the source and target domains. SSJPDA first generates substructures by clustering in the domain and then matches the substructures. The generation of substructures can dramatically reduce the data volume gap between the source and target domains. This can significantly reduce the probability of false matching. Therefore, the SSJPDA algorithm performs better than JPDA in singlemode emotion recognition. Without the influence of data volume, multimodal fusion data can provide more dimensional information to align the substructures of the source domain and target domains' substructures. Therefore, using the SSJPDA algorithm to recognize multi-mode emotional data can obtain high recognition accuracy.
The application of BPM in SSJPDA can significantly improve recognition performance. Because the emotional labels of subjects in the DEAP dataset are provided by the subjects themselves, this will affect the consistency of the emotional labels of different subjects. At the same time, because the emotional stimulation of the DEAP dataset depends on multimedia clips, some subjects also have the problem of weak emotional stimulation. In this experiment, the source domain contains all the test samples of 31 subjects, so there must be many abnormal samples and noise in the source domain. If no substructure is generated in the source domain and the data is projected directly, the abnormal samples and noise greatly impact the projecting matrix. Therefore, the advantages of the BPM algorithm are not reflected in JPDA. However, the substructure algorithm can cluster the noise or outliers of samples into the substructures of adjacent samples to reduce the impact of noise and outliers. When the source and target domain samples are clustered into substructures, we fully consider the distribution differences between the source and target domain substructures. Projecting the substructure through two different projecting matrices can better project the substructure of the source domain and the target domain to the feature space to improve the algorithm's recognition performance.

The comparison of different algorithms
When the data distribution of the training set and test set is inconsistent, the traditional machine learning algorithm cannot be competent for classification. Therefore, the 1-NearestNeighbor (1NN) algorithm performs worst in this emotion recognition problem. The purpose of transfer learning is to solve the inconsistency between the data distribution of the training set and test set, that is, the inconsistency between the distribution of the source domain and target domain. Therefore, the transfer learning algorithm performs well in this emotion recognition problem. Among them, transfer component analysis (TCA) assumes that if the marginal distributions of the source domain and the target domain are close, the conditional distributions of the two domains will also be close. Therefore, TCA projects the source and target domain data together into a high-dimensional reproducing kernel Hilbert space. In this space, the data distance between the source and the target is minimized, while their respective internal attributes are preserved to the greatest extent to complete the transfer learning. The joint distribution adaptation (JDA) method simultaneously assumes that the marginal and conditional distribution of the source and target domains are different. Then the two distributions are adapted together to achieve transfer. The goal of JDA is to reduce the distance between the source and target domain's joint probability distribution to complete the transfer learning. Balanced distribution adaptation (BDA) is improved on the basis of JDA. BDA assumes that marginal distribution adaptation and conditional distribution adaptation are not equally important. BDA adaptively adjusts the importance of marginal and conditional distribution in the distribution adaptation process according to specific data fields to complete the transfer.
We proposed the SSJPDA algorithm can better measure the distribution difference between the two domains through the joint probability distribution. This is better than JDA and BDA algorithms, which directly calculate the sum of marginal probability and conditional probability distribution differences between the two domains. In the SSJPDA algorithm, the algorithm's transferability is achieved by minimizing the difference in joint probability distribution between different domains of the same class, and the algorithm's discriminability is achieved by maximizing the difference in joint probability distribution between different domains. At the same time, using substructures reduces the difference in data volume between the source domain and the target domain and reduces the impact of noise or outliers. After using the substructure, the SSJPDA-BPM algorithm we proposed fully considers the distribution difference between the substructure of the source domain and the target domain and projects the substructure through two different mapping matrices to improve the performance of the algorithm further. Therefore, this paper's SSJPDA (BMP) algorithm has the highest recognition performance accuracy.

Discussion on negative transfer
Negative transfer means that the knowledge learned in the source domain has a negative effect on the learning in the target domain. When the source domain data is not similar to the target domain data, or the source domain data is similar to the target domain data, but the transfer learning method is not good enough that no transferable components are found, the negative transfer is likely to occur in those two cases (Pan and Yang, 2009). In this experiment, the distribution of source domain data and target domain data are different. Through the multisource domain transfer method, the data in the target domain is correctly classified by using the knowledge learned from multiple source domains so that the target domain can learn more comprehensive feature information. This can well avoid the negative transfer caused by the low correlation between the source domain and the target domain in the single source domain transfer.
However, if the source domain data used in the transfer learning algorithm contains a lot of noise, it is likely to negatively impact the classification model. The multiple source domain transfer method will further amplify the impact of noise. Regrettably, the four physiological signals, especially EEG signals, in this experiment contain numerous noise and abnormal samples. Therefore, the noise and abnormal samples in the source and target domains will inevitably lead to negative transfer. Therefore, in addition to SSJPDA-BPM algorithm, the classification accuracy of every algorithm in some subjects is lower than the chance level of 50% for two classes.
Compared with other algorithms, SSJPDA and SSJPDA-BPM generate substructures in the source domain and target domain. These substructures can properly process the data according to the data's similarity, which can validly reduce the negative impact of noise and abnormal samples in the source and target domains. It can effectively avoid negative transfer and improve the performance of the transfer learning algorithm. At the same time, as traditional migration learning methods, TCA, JDA, and BDA algorithms have a better effect on the transfer of feature size within a certain threshold. The information redundancy caused by too large feature vectors makes the impact of confusing information greater than that of task-related information, resulting in negative transfer . However, SSJPDA and SSJPDA-BPM can filter abnormal samples affected by confusing information through substructure, which further improves the algorithm's performance.
More than that, how to transfer the components found in the source and target domain data also affects the negative transfer. In comparing whether to use the BPM algorithm, if the algorithm finds the correct transferable components, projecting the effective data to the feature space through two different projecting matrices can improve the algorithm's performance and better avoid the negative transfer. However, suppose there is a lot of noise and outliers in the data. In that case, the BPM algorithm changes from an excellent method that avoids more negative transfers to a lousy method that leads to more negative transfers.

Conclusion
This paper proposes SSJPDA and SSJPDA-BPM algorithms to use the labeled physiological data to recognize the emotion of new subjects. We also explored single-mode and multimodal data's influence on emotion recognition based on physiological signals. The performance of the SSJPDA-BPM algorithm is verified by the comparative experiments of various algorithms on DEAP dataset. The results show that SSJPDA and SSJPDA-BPM algorithms can better deal with noise and outliers in data by clustering substructures. Meanwhile, these algorithms can reduce the quantity of data that better use the multidimensional information provided by multimodal fusion data. BPM algorithm can project the substructure through two different projecting matrices, which can better project the source domain and target domain data to the feature space, to improve the algorithm's recognition performance. The experimental results show that the average recognition accuracy of the proposed SSJPDA-BPM algorithm in the multimodal fusion physiological data is 63.6 and 64.4% in valence and arousal, respectively.

Data availability statement
Publicly available datasets were analyzed in this study. This data can be found here: http://www.eecs.qmul.ac.uk/mmv/ datasets/deap/.

Author contributions
ZF, XH, and JH: conceptualization and supervision. BZ, YL, and HW: data curation. ZF, XH, and BZ: methodology, writing -original draft, and review and editing. BZ, HW, and ZF: validation. XH and YL: visualization. All authors contributed to the article and approved the submitted version.


s2fieldsofstudy:
Computer Science
Engineering

text:
A framework of safety analysis with temporal feature based on MBSA and case study for ACC system

. The safety of automotive Adaptive Cruise Control (ACC) system is of great significance to prevent fatigue driving, improve driving comfort, reduce accident rate and promote the development of intelligent transportation and au-tonomous driving technology. However, the current safety analysis of ACC lacks consideration of the temporal dynamic property, so it is necessary to establish a set of safety analysis methods to consider the temporal characteristics. This paper proposes a new safety analysis method based on MBSA framework and intro-duces temporal features. Altarica3.0 is a high-level modeling language for safety analysis, and its basic mathematical form is Guardian Transformation System (GTS). In this paper we outline an analysis approach that converts failure behav-ioral models (GTS) to temporal fault trees (TFTs), which can be analyzed using Pandora a recent technique for introducing temporal logic to fault trees. However, like classical fault tree analysis, TFT analysis requires a lot of manual effort, which makes it time consuming and expensive. In order to improve the safety of the system, the proposal extends Bayesian Networks with Pandora and results to dependability analysis with temporal relationships to provide more reliable basis for safety design. As a typical case study, the safety analysis method proposed in this paper is applied to the safety analysis of adaptive cruise system, and the results show the effectiveness of the proposed method. Furthermore, it also pro-vides new technologies for the automation and intelligence of safety analysis for smart internet of vehicle.

for Basic Requirements of Software Safety Assurance), GB/T34590.10-201 (Functional Safety of Road Vehicles), ARP4761 (Criteria and Methods for safety Assessment process of civil airborne systems and equipment) [2], etc. Strict compliance with safety standards at all stages of system design can greatly reduce the probability of system failure.
In the process of System Safety Assessment, the main steps include Functional Hazard Analysis (FHA) [3], Preliminary System Safety Assessment (PSSA) [4], and System Safety Assessment (SSA) [5]. Traditional safety analysis methods mainly include fault tree analysis (FTA) [6], Markov analysis (MA) [7], failure mode and effect analysis (FMEA) [8] and common cause analysis (CCA), etc. But with the passage of time, the traditional methods of risk assessment of excessive dependence on low level form of modeling, system specification and related models between cannot match very well, when very small changes in the system specification, the relevant safety model also need to conduct a comprehensive assessment, a lot of manpower and material resources, at the same time.
When the analyst converts the actual physical model into the corresponding fault tree, there is a great subjective influence, and there may be the consequence of inaccurate modeling.
To solve these problems, a model-based safety analysis (MBSA) method is proposed. It is actually the reliability Engineering branch of Model-based System Engineering (MBSE) [9,10]. Model-based system safety analysis can be divided into two steps: system modeling and related safety analysis for the model. To be specific, the first step is to write the system model in the high-level modeling form, which can completely express the function and structure of the original system. The second step is to evaluate the model directly, or compile the model into a low-level model for evaluation, such as FTA, MA, etc.
AltaRica is a kind of safety for fault modeling language, to the guard conversion system (GTS) as the core [11][12][13]; the use of reusable object-oriented language description node and fault through the node connected to the interface between embedded systems describes the interaction between the node and system information, including data flow elements support interaction between components, there are dependencies between components.
AltaRica3.0 is a new version of the AltaRica language [14]. Its new underlying mathematical model is the guardian transformation system (GTS) [15], which improves the expression ability of previous versions by introducing a fixed point mechanism to stabilize the theological value after each transition. This mechanism allows the design of causal components and the handling of systems with immediate cycles. In the framework of AltaRica3.0, this paper proposes an automatic construction method of Pandora timing fault tree based on extended fault tree model. The method is verified by modeling and simulation of typical structural model of ACC system. After that, quantitative analysis of Pandora timing fault tree was carried out. Since traditional analysis methods are prone to state explosion and fault events are interdependent, reliability analysis of Pandora temporal fault tree was carried out in this paper based on Bayesian network. The Pandora temporal fault tree analysis and probability calculation provide a more reliable basis for the safety of system and give system designers an idea where to invest more design efforts to improve safety. Finally, the design defects of the system design model are tracked by combining the verification results and traceable information model.
To elaborate on the composition of the GTS, Fig. 1 shows a graphical representation of a cooling system and its flattened model representation.
Concrete components in the cooling system are abstracted into classes: tank, pump and reactor, respectively. These classes interact with each other through variables. Tank instantiates object T outflow coolant as outflow variable of the whole system, and two instantiated objects P1 and P2 of pump have two variables, respectively: outflow from tank, pump and reactor. Inflow of input variable, inflow of reactor is the same as outflow of P1 and P2. The system has to keep at least one pump running so that the coolant can be used to cool the reactor properly, but the pump could fail or the tank could be empty, which could cause the entire system to fail.
The GTS model obtained by flattening the hierarchical information of the cooling system is shown in Fig. 2.

Pandora temporal fault tree
Pandora defines three time gates: Priority-AND gate (PAND), Priority-OR gate (POR) and Simultaneous-AND gate (SAND) to extend the classic fault tree [17]. The temporal fault tree symbols of the three gates are shown in Fig. 3. PAND gates are not new and have been used in FTA since the 1970s [18]. The symbol " < " used to represent the PAND gate in the logical expression, such as A < B represents A PAND Y, where both A and B are failure events. the Priority-OR (POR, symbol '|'), which as described earlier models a priority situation where one event must occur first and other events may or may not occur subsequently. The Simultaneous-AND (SAND, '&') gate, which represents simultaneous occurrence of events, e.g., as a result of a common trigger. In this article, we use " + " for OR gate and "." for the AND gate. Pandora allows more than one form of reduction. As well as removal of redundancies, e.g., A < B + A|B ⇔ A|B (where A and B are fault tree events), Pandora also allows reduction via the recognition and removal of contradictions and by means of 'completion'-conversion of temporal expressions into static Boolean expressions. For full details about Pandora's temporal laws, we refer the reader to [18], but four laws in particular will be useful:

Bayesian network
Bayesian network (BN) is a directed acyclic graph, which is composed of nodes and directed edges. Nodes represent variables and directed edges represent relationships between nodes. A BN model contains a qualitative and a quantitative part (see Fig. 4). Bayesian network is a directed acyclic graph representing a group of random variables and their conditional correlation [19]. The node of BN represents the random variable, and the directed arc represents the dependence or causality between nodes. In the Bayesian network, if there is an arc from node X and node Y, node X is said to be the parent node of another node Y. Has direct influence to the subsidiary to the parent node, it is defined as a Pr {Xi | parent (Xi)} to quantify the influence of the subsidiary to the parent node. A node is a root node if it has no parents, and a leaf node if it has no children. Discrete-time Bayesian network (DTBN) is a BN with N nodes is generally represented N = < < X, T n , E > , P > . The nodes X = {X 1 , …, X N } in the figure represent variables, and all phenomena such as component state values and personnel operations can be abstracted into the node variable. T n = {[t 0 , t 1 ), …, [t n − 1 , t n ), [t n , ∞} is a division of the mission, its elements represent the interval where the variable fails at the time. Directed edge E characterizes the causal relationship between node variables. In (X i , X j ), X i is the parent of X j , and X j is the child of X i . A node without a parent is a root node, and a node without children is a leaf node. Digraphs imply conditional independence. P represents conditional probability distribution (CPD). It can be seen from the implied independence assumption of the Bayesian network that the joint probability distribution including all nodes can be obtained on

A framework for automatic generation of Pandora temporal fault tree based on GTS
In general, this section gives the method framework and several steps of how to automatically generate the Pandora temporal fault tree based on the flattened GTS model of Altarica3.0 as shown in Fig. 5. The algorithm GTS to Pandora temporal fault tree (GTS2PTFTA) design involved in each step is then explained in detail: (1) Read the flattened GTS model description file and construct the corresponding GTS object. (2) The input GTS object is divided into a set of independent GTS and an independent assertion. (3) Iterate the independent GTS set to obtain the corresponding reachable graph (state machine graph) of each independent GTS, so as to facilitate the subsequent generation of event sequences with timing characteristics according to the state machine graph design algorithm. (4) Compile each reachable graph and obtain the logical formula of the independent GTS with the temporal relationship. Find all possible paths between two nodes in the reachable graph and design the algorithm to update the possible priority relationship into the path. The algorithm described in Step 4 is corresponding to the logical formula containing time sequence relation of the 3.3 sub-GTS model reachable graph. The AND gate operator is further annotated as: search all paths from state S0 to the other states of the graph. Events occurring along path π are transformed into the relationship of events. Each state of the graph is first associated with a sequence disjunction obtained through the compile path, then each pair (variable values) associated with a sequence of disjunction, including variable uses this value, secondly, traverse to search path, looking for the same end node status on different paths to share events, update priority or temporal sequence in the original sequence of disjunction, after exhaustive recursion and, at the same time, and preference or conversion between matching rules, generating a sequence of events with a temporal relationship. (5) Synchronize the independent assertions obtained by partition with the formulas obtained by compilation of each independent GTS reachable graph and obtain the set of temporal relational event sequences of the whole model, which is composed of the Pandora temporal fault tree of this GTS model.

GTS model preprocessing and partitioning
A partition operation on a GTS instance object. Considering the large scale of the system model, after the GTS instance object is obtained, in order to simplify the subsequent steps, a more efficient partition algorithm is adopted here to improve the efficiency of the algorithm, so as to cope with the large scale of the system model. The partitioning algorithm divides a model into multiple components and modules, and then processes each component and module individually. When the fault tree with time sequence relationship is finally obtained, it is processed together with the data results, which simplifies the intermediate steps of the whole algorithm framework process.

Construction of the accessibility diagram of the sub-GTS model
The corresponding reachable graphs of each independent GTS were obtained. In this step, we mainly process each independent GTS to obtain the relevant reachable chart. A reachable graph contains a set of nodes, and a node exists in the form of a set of variables, and the current system state is represented by the values of the current variables.

Algorithm for compiling reachable graph to generate temporal expression of sub-GTS model
An accessibility graph is a state machine with a finite number of states. It may change state as an event occurs, but at each moment, it is only in one state. In the previous step, the algorithm obtained the reachable graph of each GTS. To obtain a logical formula containing temporal relationships, in short, find all possible paths between two nodes in the reachable graph and design an algorithm to update the possible priority relationships to the paths. The algorithm mainly uses priority or gate (POR) to distinguish fault sequences. A gate can represent a situation in which one event takes precedence over others and must occur first, but does not specify that other events must also occur, for example, if A occurs and B does not occur, or if A occurs and B occurs but A occurs first, then POR B is true. In this step, operator "." is defined as a sequence of "and" door and operator "+" is defined as a sequence of "or" door and operator "<" is defined as a sequence of "priority-AND gate," operator "|" is defined as "priority-OR" gate and the operator "ø" represents a collection of sequences. The AND gate operator is further labeled as search all paths from state S0 to other states of the graph. Events occurring along path π are converted to and relationships of events. Each state of the graph is first associated with the sequence disjunction obtained through the compile path, then each pair (variable values) associated with a sequence of disjunction, including variable uses this value, secondly, traverse to search path, looking for the same end node status on different paths to share events, update priority or temporal sequence in the original sequence of disjunction, after exhaustive recursion and, at the same time, and preference or conversion between matching rules, Generates a sequence of events with timing relationships.

Definition 2 Reachable Graph (RG).
A reachable graph is a quadruple (S, Σ, δ, s 0 ) where: • S is a finite set of states.
• Σ is a finite set of events, such that S ∩ Σ = ∅.
• δ is a partial function: S × Σ → S, s.t. for (u, u') ∈ S 2 , and e ∈ Σ, u' = δ(u, e) iff e is incident from u to u', and we write it as: u → u'(e). • s 0 is the initial state.

Definition 3 Paths Set.
Let P be the set of all paths in the RG,

Definition 4 Forward and backward incidence sets.
For any state u ∈ S, let ΣuI (resp. ΣuJ) be the set of events incident from u (resp. incident to u),

Definition 5 Set of final states.
Let F be the set of the final states, F = {f ∈ S| ΣfI = ∅}.
The algorithm and pseudocode of compiling the accessible graph of independent GTS to obtain the time-series relation formula are as follows: Generates a set Φ of Pandora formula: Φ = {φ s |s ∈ F} one formula φ s for each final state s. These expressions can then be analyzed by Pandora. Transformation algorithms are biased toward increasingly dynamic systems. The best case complexity for checking the necessity of chronological order is O(n), and the worst case complexity is O (n 2 ), where n is the number of paths from the initial state to the final state in the reachable graph. In the best case, for each divergent path, there exists a shareable event that is related to the direct reachable state of the connection state when the path diverges. In this case, the cost of adding j to each state is the order time of an O (m 2 ) operation, where m is the degree of j.

Safety analysis of Pandora temporal fault tree based on Bayesian network
To perform quantitative analysis of temporal failure behavior, Bayesian Network has been integrated with the Pandora temporal fault Tree, a compilation of the AltaRica model-based dependability analysis process.
The primary goal of the Bayesian network-based approach is to transform the Pandora TFT into a Bayesian network for quantitative analysis of dynamic systems using BN models. The transformation from TFT to BN is done in two steps. In the first step, the TFT to BN graphic mapping is achieved by converting the base event to the root To represent sequence values, which are used in Pandora to represent sequencing among events, in this paper, the mission time T is divided into N equal intervals from t = 0 to t = T, where each interval represents a possible non-zero sequence value, during which an event occur. Initially, all components are assumed to be fully operational and therefore all events are given an initial sequence value of 0 (i.e., the component has not yet failed). This value holds until the occurrence of a component failure, i.e., if the component can survive till the end of the mission time then it will continue carrying the sequence value 0. If a component fails in interval 1, then it will have the sequence value 1, but if the component fails in any other interval i where 1 < i < N then it can have any sequence value in between 1 and i based on its relative position to its immediate predecessor. An event having a sequence value i is considered to be in State i and it has an associated probability value representing the probability of the event being in State i. As the root nodes in the BN represent different basic events, we need to define prior probability tables for the root nodes, where each entry in a prior probability table of a node represents the probability of the respective event being in a particular state. For exponentially distributed failure rate, the probability of a component being failed in the interval [t1, t2] (e.g., in State i) can be obtained by integrating the probability density function of exponential distribution, λe, λt, in the following way: Once prior probability values are assigned to each root node, conditional probability values are generated for all the intermediate nodes. Remember that Pandora represents the outcome of every gate with a sequence value, showing not only whether that gate is true or false but also the relative order in which the gates become true. This is purely deterministic: the outcome of each gate depends solely on the sequence values of its input events. Consequently, the probability of an intermediate BN node (representing a gate) being in a certain state can either be 0 or 1, depending on the state of its parent nodes. In [20], the authors have shown the translation and CPT generation process for the PAND gate, as seen in Fig. 6. The mapping of a POR gate by diving the mission time into two intervals is shown in Fig. 7. The CPT of the PAND gate resembles its temporal truth table. As seen in the CPT of the PAND gate, the PAND outcome becomes true only in one scenario when the first input (X) is in State 1 and the second input (Y) is in State 2, i.e., they occur in a sequence from left to right. In this case, the state associated with the PAND outcome is State 2 because this is the state when the last input of the PAND becomes true. Bayesian Network of the AND and OR gates are shown in Figs. 8 and 9, respectively. From the CPT of the AND gate, we can see that the AND outcome becomes true (1s in column State 1 or State 2) when the input events are either in State 1 or State 2. If any of the input event is in State 0 (logical false), then the outcome of the gate is false (1s in the column State 0). So the CPT of the POR, AND and the OR gates are of the same pattern and entries in the table are either 0 or 1, but the positions of the 0s and 1s change according to the logical specification of the gates. Once we have the Pandora TFT of the failure behavior of a system, we can translate the TFT to BN model and subsequently perform predictive reasoning on the Bayesian Network to obtain system unreliability. This is done by following the direction of the BN arcs from the root nodes toward the leaf node. In this process, failure probability data of the root nodes are used to obtain the probability of system failure, i.e., data about causes are used to obtain new belief about the effects. Using the facility of observing the status of a node, we can also perform diagnostic analysis on the BN, i.e., reasoning from effects back to their causes. If the analysts have the evidence that the system has failed, then based on this evidence the analysts belief about the failure probability of the components can be updated. That means we now have to put an observation on the leaf node of the BN and work backwards (in the opposite direction of the BN arcs) toward the root nodes to update the probability of root nodes. In order to facilitate diagnostic analysis, Bayes' theorem in Eq. (1) is used.

Description of ACC system
Typical ACC system of automobile is mainly composed of information perception unit, electronic control unit (ECU), execution unit and human-computer interaction interface, etc. This paper selects some of these structures as examples for analysis, and the basic composition is shown in Fig. 10. In the ACC system, ranging radar used to measure the vehicle and the vehicle in front of the car distance, relative velocity and relative acceleration, is one of the key equipment of adaptive cruise control system, is to determine the main components of the cost of the system, its main components including transmitting antenna and receiving antenna, DPS (digital signal processing (DSP)) processing unit, data cables, etc. The electronic control unit in ACC module is composed of large-scale integrated circuits, such as microprocessor (CPU) memory (ROM RAM) input/output interface (I/O) analog-to-digital converter (A/D) and shaping driver, as well as common single-chip microcomputer, according to its memory program and data on the air flow meter and a variety of sensor input information processing judgment, and then output instructions.
The main working principle of the ACC system is Lord radar sensors to detect the target vehicle in front of the car, and to provide the main vehicle and target vehicle electronic control unit (ECU) the relative speed between the relative distance information such as the relative azimuth Angle Electric control unit based on the pilot set the safety car distance and cruising speed, combined with radar transmit information to determine the main vehicle driving state, issue instructions to adjust throttle opening and brake control of main vehicle. The arrow in the figure indicates the direction of signal transmission.

System modeling and analysis of ACC system
Model-based analysis of system is performed in a compositional fashion. For this reason, the ACC system is divided into smaller subsystems and the subsystems are further divided into smaller subsystems to the level of components. Components are annotated with their failure behavior and their behavior are combined to obtain the behavior of the subsystem possessing them. We will use this system structure as an example to describe the different steps involved in compiling the Guard transformation system into a Pandora temporal fault tree.  The GTS, describing a ranging sensor, is given in Fig. 11 (Set as secondary redundancy component, S1 as the primary component and S2 as the backup component), automobile sensor is similar to: Monitor component modeling: The monitor is mainly used to monitor the output of the main component. If any omission or failure is detected, the backup component will be activated, as shown in Fig. 12.
The GTS, describing an electronic control unit, is given in Fig. 13. Brake pressure module and throttle opening module are the same as above. Modeling of the whole adaptive cruise system is shown in Fig. 14.
The whole system model was analyzed: First, the AltaRica3.0 model obtained was flattened; second, the reachable graph of each independent GTS was obtained according to the correlation partition algorithm, as follows: The reachable graph for the remaining components is similar to the figure above. "O-x" represents the output omission or fault of a component, "X_Failure" represents the failure event of a component, the box represents the status of the component at this time, and the line between the boxes represents the transmission of failure events. For descriptive purposes, any part symbol abbreviates the failure of the corresponding part (e.g., "M1" is denoted as "M1 failure").
The TFT of component S1 is the simplest: O − A = A (i.e., failure of A causes omission of A), as shown in Fig. 15. For the reachable graph of M1 in Fig. 16, the initial state is "WORK-ING," which means the sensor is ready to activate B. If a failure occurs in the output of S1 (for example, an output omission represented as "O-S1" in the reachable graph) and is detected, the state of M1 changes to "omission detected" (that is, B needs to be activated). However, if M1 fails after waking up S2 (because S2 takes over the functions of S1), the system can continue to operate, so the failure sequence (O-S1 first, M1 second failure) will cause the sensor to enter the "safety failure" state. Conversely, if M1 fails first (into a "premature failure" state), it will not be able to detect subsequent output omissions from S1, so the second sequence will result in a "serious failure" state (since B will not be activated, which means the system is completely faulty). The latter state represents an output omission from M1 (" O-M1 "), where errors propagate through the system (in this particular case leading to complete failure). Therefore, as you can see, from the situation in Fig. 15 O-S1 Fig. 15 Reachable graph of S1 of the sensor. One sequence can cause the system to fail, while the other gives it a chance to continue working during backup. Therefore, we will introduce Pandora gates to represent the sequence of events in the fault tree, and then the reachable graph can be transformed into a Pandora temporal fault tree without sacrificing the relevant event sequencing, which affects the outcome. So the above two sequences are converted to: If we apply this algorithm to the reachable graph of component S2, as shown in Fig. 17, we get the following fault state expression:

Generate Pandora temporal fault tree
According to the automatic generation algorithm introduced in Sect. 3, the sequence of timing fault tree is as follows: Therefore, the resulting Pandora timing fault tree is shown in Fig. 18.

Model-based analysis of the ACC system
The ACC system is analyzed based on the model to obtain the cause of failure. The temporal fault tree obtained from the analysis is shown in Fig. 18. As you can see from TFT, some basic events are shared with each other, which makes them statistically dependent. If we use an analytical approach to quantify TFT, statistical dependencies between events are ignored. However, the proposed method of converting to a Bayesian network can quantify TFT by taking these dependencies into account, resulting in more realistic results.  To conduct quantitative analysis of the TFT, the Pandora TFT is mapped to the Bayesian network model, as shown in Fig. 19. The failure rate of Pandora TFT basic events is shown in Table 1, and time changes are characterized by the failure rate of basic events.
The prior probability table of each root node of BN is filled according to the failure rate of the corresponding component. As mentioned in Sect. 4, in order to generate the prior probability value of the event, we need to define a task time and interval number (N), and divide the task time into discrete intervals. Assuming that the task time is 5000 h, the task time is divided into four intervals (N = 4), and the prior probability value of the root node of BN is calculated. The conditional probability table for each intermediate node in BN is filled according to the type of TFT gate represented.
The reliability analysis of the ACC system is carried out based on the DTBN model. Compared with the results of the Markov chain method, the results are shown in Table 2. It can be seen that the DTBN method has a high solution accuracy and Markov chains cause state explosion problems. Reliability analysis of ACC system is done by running queries on BN models. Meanwhile, using BN-based method, it can also be concluded that ACC is the most critical basic event of TFT, followed by Si (i = 1,…, 4). This gives the system designer an idea of where to put more design effort to improve the reliability of the system. For example, an event ACC corresponds to a failure of the core electronic control within the system, while Si corresponds to a failure of the ranging sensor in the system. Therefore, if the analyst wants to improve the reliability of the system, then they can consider replacing the above key components with more reliable components, or they can consider introducing redundant components along with the key components. By observing the BN node endpoint in Fig. 19, in the failed state, the system was diagnosed and analyzed. This describes a scenario in which the analyst has evidence that the most significant event of the TFT has occurred. Based on this evidence, analysts' beliefs about prior probabilities of fundamental events have been updated. Based on these updated beliefs about the failure probability of basic events, a whole new predictive analysis can be performed.
By observing the leaf node of the BN model in Fig. 19, in the failed state, the system was diagnosed and analyzed. This describes a scenario in which the analyst has evidence that the most significant event of the TFT has occurred. Based on this evidence, the belief of the analyst on the prior probability of the basic event has been updated, and the posterior failure probability of the basic event is shown in Fig. 20. Based on these updated beliefs about the failure probability of basic events, a whole new predictive analysis can be performed. A posterior failure probability curve explains the dynamics of system failure and provides updated causality between variables that may change over time.

Conclusion
In recent years, more and more attention has been paid to the research of automatic fault tree generation. However, considering the timing fault characteristics of the system, AltaRica language modeling and automatic generation of timing fault tree algorithm in the field of the Internet of vehicles has not made much progress. This paper is based on the AltaRica model and studies how to automatically generate Pandora timing fault tree according to its flattening results GTS. The algorithm in this paper is a comprehensive design and implementation. First, the independent GTS of the whole model is obtained by dividing the algorithm, and then it is stored in two ways of edge node set and variable state node set in the module to obtain the reachable graph, and then expressed by adjacency matrix. The benefit of this approach is that it is fault-tolerant and uses different storage methods to store the information in the diagram, so that it can be verified in many aspects later when facing large system models. In compiling the accessibility graph algorithm, each TFT obtained is aggregated upward and unified into a system TFT, and assertions are propagated when multiple state results have been obtained, which is much more efficient than directly analyzing a complete GTS to generate a fault tree. Then, Pandora TFTs are generated and combined with Bayesian networks for reliability analysis, so as to realize effective automation of reliability analysis of complex dynamic systems. In the following work, we will further combine the previous work with the current to build a complete system safety analysis for AltaRica3.0, and carry out a large-scale application case study in the typical field of smart Internet of vehicles.