Deep Reinforcement Learning Assisted Federated Learning Algorithm for Data Management of IIoT

The continuous expanded scale of the industrial Internet of Things (IIoT) leads to IIoT equipments generating massive amounts of user data every moment. According to the different requirement of end users, these data usually have high heterogeneity and privacy, while most of users are reluctant to expose them to the public view. How to manage these time series data in an efficient and safe way in the field of IIoT is still an open issue, such that it has attracted extensive attention from academia and industry. As a new machine learning (ML) paradigm, federated learning (FL) has great advantages in training heterogeneous and private data. This paper studies the FL technology applications to manage IIoT equipment data in wireless network environments. In order to increase the model aggregation rate and reduce communication costs, we apply deep reinforcement learning (DRL) to IIoT equipment selection process, specifically to select those IIoT equipment nodes with accurate models. Therefore, we propose a FL algorithm assisted by DRL, which can take into account the privacy and efficiency of data training of IIoT equipment. By analyzing the data characteristics of IIoT equipments, we use MNIST, fashion MNIST and CIFAR-10 data sets to represent the data generated by IIoT. During the experiment, we employ the deep neural network (DNN) model to train the data, and experimental results show that the accuracy can reach more than 97\%, which corroborates the effectiveness of the proposed algorithm.

I. INTRODUCTION
Human society is rapidly moving towards the era of industry 4.0 [1]. The global distribution of user equipments (UEs) is widespread and decentralized due to the influence of geographic location. Limited by the costs of transmission media (cables, optical fibers) and communication delays, traditional network infrastructures are not suitable for the development of Industry 4.0 [2]. On the other hand, wireless networks are widely used in driverless cars, smart grid and smart medical [3]. Fig. 1 shows the IIoT scenario under the background of rapid development of social science and technology. A large number of IIoT equipments access to IIoT frequently, which can produce a huge amount of data in a short period of time [4], [5]. Radio network resource management faces severe challenges, including storage, spectrum, computing resource allocation, and joint allocation of multiple resources [6], [7]. With the rapid development of communication networks, the integrated space-ground network has also become a key research object [8]. How to efficiently manage, store and use these time series data has become an important research topic.
The data generated by IIoT equipments often involves users' private information [9], [10]. Service providers and users do not want this information to be exposed to the third party, but this kind of data is usually vulnerable to attacks from heterogeneous networks, heterogeneous equipments or malware [11]. Therefore, there needs an approach to support IIoT, that cannot only maintain the heterogeneity and privacy of data, but also reduce the communication cost and model training deviation [12], [13]. As a new type of machine learning (ML) paradigm, federated learning (FL) has attracted great attention in the industry [14]. FL prevents the leakage of users' personal private information to a certain extent by separating the central server's direct access to the original data from the model training [15], [16]. In addition, FL can effectively maintain the heterogeneity of data and reduce the deviation of model training. Reference [17] has proved that FL can be effectively applied to multiple learning tasks. Considering the privacy, heterogeneity and wide distribution of the device data in IIoT, FL in IIoT may have a positive effect.
With the rapid popularization of IIoT technology, the geographical distribution of IIoT equipments is becoming more and more extensive and the equipment forms are quite different from each other [18]. This leads to a series of problems such as uneven data quality of equipment and high communication cost. In order to increase the model training rate of IIoT equipments and reduce the communication cost of model aggregation, driven by the current artificial intelligence (AI) technologies, we propose a deep reinforcement learning (DRL) assisted FL framework [19], [20]. DRL algorithm is suitable for solving decision-making problems in high dimensions [21], [22], so it can use the decision-making effect of DRL to select some high-quality local IIoT equipment models for aggregation. Considering that a large amount of data generated by IIoT equipments will lead to excessive training data, increase the burden on wireless network channels and may cause privacy leaks, we employ a distributed method to train  DRL agents, i.e., apply the DRL assisted FL method on each IIoT equipment to realize the effective application of FL in IIoT. The main work of this paper is as follows: 1) Aiming at the problem of difficult management of IIoT devices due to the generation of large amounts of heterogeneous and private data, this paper adopts a FL method to train and manage these data. 2) In order to improve the performance and efficiency of FL, this paper proposes a DRL-assisted FL framework. DRL algorithm based on deep deterministic strategy gradient (DDPG) is mainly used to select IIoT device nodes with high data quality, so as to increase the model aggregation rate and reduce communication costs. 3) We analyze the characteristics of IIoT device data, and then use MNIST, Fashion MNIST and CIFAR10 (IID and non-IID) data sets for experimental evaluation. Finally, we verify the effectiveness of FL technology based on DRL to process IIoT data.
The remaining part of this paper is organized as follows: Section II reviews the work related to the application of AI technology in IIoT. Section III introduces in detail the related issues of the application of FL in IIoT. Section IV introduces the realization of FL algorithm assisted by DRL and the node selection process of applying DRL algorithm. Section V describes the experimental setup of FL algorithm assisted by DRL and shows the experimental results. The last section concludes the paper.

A. Related Work of Industrial Internet of Things Based on Deep Reinforcement Learning
IIoT has been widely concerned by the industry. Especially in the field of IIoT, there have been many solutions to solve the practical problems of IIoT by using AI technology. Due to the explosive growth of user equipments and data streams, IIoT has seen a shortage of spectrum resources. Shi et al. [23] proposed a spectrum resource management scheme for IIoT networks. Specifically, in order to consider the differentiated communication needs of different users, the authors proposed a modified deep Q-learning network (MDQN) and designed a new reward function to drive the learning process. Afterwards, the authors established a simple medium access control model, which used the base station as the sole agent to manage spectrum resources. In the end, the solution promoted the sharing of spectrum between different types of users. Chen et al. [24] studied the joint power control and dynamic resource management of multi-access edge computing (MEC) in IIoT. The authors transformed this problem into Markov decision process (MDP) and used the dynamic resource management algorithm based on DRL to solve this process. The algorithm fully considered the dynamic and continuity of task generation, and finally used the DDPG to optimize the long-term average delay of dynamic resource management. The experimental results showed that the method was effective. Based on blockchain technology, Liu et al. [25] proposed an emerging data collection and sharing scheme to deal with the two main problems faced by IIoT. The first was to achieve efficient data collection within the limited energy and sensing range of mobile terminals. The second was to ensure the security of data sharing between mobile terminals. The authors combined blockchain and DRL algorithms, used DRL to maximize data collection, and then used blockchain technology to ensure data security. This scheme embodied the superior performance of the two technologies.

B. Related Work of Industrial Internet of Things Based on Federal Learning
The latest development of FL in IIoT was an efficient deep anomaly detection framework based on equipment joint learning proposed by Liu et al. [26], which was used to detect time series data in IIoT. The authors first used a FL framework to enable edge equipments to co-train an anomaly model. After that, they used the attention mechanism-based convolutional neural network (CNN) long short-term memory (LSTM) model to capture important fine-grained features. In the end, the authors used a gradient compression mechanism based on Top-k selection to improve communication efficiency. A large number of experiments on real data sets showed that the algorithm can effectively reduce communication overhead. FL was originally proposed by Brendan et al. [27]. They first proposed a decentralized method for private data training-FL, and then conducted extensive experimental evaluations using five different model structures and four data sets to verify the robustness of the FL method. This research opened a precedent for the exploration of FL technology. In addition, there were some typical representatives who applied FL to solve different actual network problems [28]- [32], which had shown good results in solving the computing, caching and communication, and IIoT problems of the intelligent mobile edge.
We summarize the application classification of FL and DRL in TABLE I. Similar to the above works, we have jointly paid attention to the data privacy issues in the IIoT scene, and proposed feasible solutions from the perspective of design framework and algorithm implementation. However, the biggest difference between our work and the above works is that we use DRL algorithm to select high-quality IIoT equipment nodes for FL, so as to improve the rate of model aggregation and reduce the communication cost. This is not reflected in the above work.

III. FEDERATED LEARNING APPLICATION PROBLEM DESCRIPTION IN INDUSTRIAL INTERNET OF THINGS
The core goal of FL is to carry out efficient ML among multi-UEs or multiple computing nodes under the premise of ensuring the security and privacy of data communication [33]. IIoT equipments need to upload the local model to a central server after using local data for model training, and the central server will optimize the global model. The general steps are: 1) IIoT equipments use local data for local calculation, while minimizing the predefined empirical risk function, and then update the calculated weight to the wireless network access point. 2) The wireless network access point collects the weight of IIoT equipment update and accesses the FL unit to generate the global model.
3) FL redistributes the output of model training to IIoT equipments, which use global models for a new round of local training. 4) Repeat the above steps 2 and 3 until the loss function converges or reaches the maximum number of iterations. Suppose that there are N IIoT equipment nodes in an IIoT scenario, the local data set composed of local data of each IIoT-DN is {X 1 , X 2 , ..., X N }. The IIoT equipment node downloads the global model θ from the central server and trains them by local data set alignment. IIoT equipment node uploads new weights or gradients to a central server to update the global model. Therefore, the data sample size from N IIoT equipment node is N n=1 x n = X. Then the loss function of IIoT equipment node with data set X N is where f j (θ) is the loss function of data sample j. Optimize the global loss function by minimizing the weighted average of the local loss function F n (θ) of each IIoT equipment node training sample: We summarize the loss functions of several common ML models that can be used for FL in TABLE II [34]. DRL agent training real data from IIoT equipments has more advantages than agent data provided by the data center. The data generated by IIoT equipments is usually highly secretive and large in scale. DRL agents also need to record them on a central server when they are trained based on local models. In order to better verify the effectiveness of FL assisted by DRL, we extract the data characteristics generated by IIoT equipments as follows: (1) high privacy, (2) unbalanced amount of data (a lot of user output is used, the user output is low), (3) the distribution scale is large, and (4) the user equipment is limited by the communication quality. Corresponding to the actual situation of the data generated by IIoT equipments, our experiments are carried out on MNIST, Fashion MNIST and CIFAR-10 (independent and identically distributed (IID) and non-independent and identically distributed (non-IID)). The reason for using the above data sets is that they can reflect the characteristics of data heterogeneity, scale difference and dispersion to a certain extent.
In the case of non-IID, the data between users can be divided equally or unequal. We assume that there is a fixed number of IIoT devices in each round of communication between the server and devices. When the communication process starts, the devices are randomly divided into several groups, and the server will send the current global model parameters to each device. In order to improve the communication quality, we use DRL algorithm based on DDPG to select some device nodes to participate in the training. After that, each selected device will calculate according to the global state and local data set, and then send the update to the server. The server uses this repeated method to update the global model parameters. In where vn is the weights connecting the neurons, N is the number of neurons and φ() is the activation function.
this way, the non-IID characteristics of IIoT device data can be considered. The experimental setup part will be introduced in detail later.

IV. IMPLEMENTATION OF FEDERATED LEARNING ALGORITHM ASSISTED BY DRL
In this part, we will give a FL framework assisted by DRL in section IV-A, which is the core technology to realize the data training of IIoT equipment. Based on this framework, we will describe in detail the implementation steps of the FL algorithm assisted by DRL in section IV-B. Finally, we will introduce the process of applying DRL algorithm to achieve IIoT equipment selection.

A. Framework
The purpose of integrating DRL into FL is to intelligently use the cooperation between IIoT equipments and nodes to exchange learning parameters, so as to better train the local model [35]. Due to the limited cache and performance calculation of the edge IIoT equipments and nodes, and the direct transmission of a large amount of data may cause network channel congestion or even data leakage, and so the data is trained in the local equipment. Therefore, we make full use of the efficient model training performance of DRL and conduct data confidentiality training based on FL. The final purpose is to verify the training effectiveness of equipment data in IIoT. We deploy DRL agents in every edge IIoT equipment. Considering the inherent characteristics of wireless network, if the amount of data to be trained is large, it will increase the burden on the wireless network channel. Moreover, the data needs to be transformed for privacy reasons, so the correlation of the central server-side agent data may not be as good as the data on the edge equipments. In addition, deploying DRL agents on edge equipments alone may cause additional energy consumption.
To deal with the above problems, we propose a FL framework assisted by DRL, as shown in Fig. 2. Considering a large number of data factors, we adopt the distributed deep learning (DL) method. As shown in TABLE I, distributed DL has good performance of fast training and edge efficiency. Therefore, the fast training of the data on edge equipments can avoid network congestion to certain extents. Considering that the correlation of agent data on the central server is not as good as that of the edge equipment, in the IIoT scenario, a large number of IIoT equipments are equipped with perceptron, which can obtain rich and personalized data to update the global model. Based on these data, the central server-side agent data can be effectively updated to match the data of the edge equipment side. The FL framework based on DRL mainly completes three tasks: node selection, local training and global aggregation. Node selection is achieved by using the DRL algorithm based on DDPG, in order to select IIoT nodes with high data quality to participate in the model training process. The purpose of local training is to derive local model parameters suitable for local nodes. The purpose of global aggregation is to generate a global model suitable for data training, which is achieved by uploading local model parameters to each local server.

B. Implementation Steps
The FL framework assisted by DRL has three main implementation stages: 1) Initialization stage: The central server evaluates the connection request of IIoT equipments. After that, the central server randomly selects a subset of user equipment from the connected IIoT equipments to participate in this round of training [36], [37]. After training, a global model θ t is sent to each selected IIoT equipment. 2) Training stage: The selected IIoT equipment uses local data to train the global model, and the training process is θ t → θ n t , get the global model θ n+1 t after each iteration. For the n-th IIoT equipment, the optimization objective of loss function is where F (θ) is the n-th local loss function. for the next iteration. After that, the central server issues the new global model to the newly selected IIoT equipment collection. Repeat the above process until the loss function converges or reaches the maximum number of iterations.
In the FL algorithm assisted by DRL that we designed, we select IIoT equipments with a proportion of C in each round and then calculate the gradient loss of these IIoT equipment data. For IIoT equipment n, calculate the gradient g n = ∇F n (θ t ) under the current model θ t and then aggregate these gradients by the central server. Use the following formula to update: The operation of selecting specific IIoT devices is called a scheduling strategy. Since the reading and execution of data in the central server and the reading and execution of data in various IIoT devices are not in the same order of magnitude, uploading the training and updated models of all IIoT devices to the global controller will cause a lot of computing time and communication overhead. Therefore, it is an effective way to select a specific part of equipment upload parameter model through the scheduling strategy. Scheduling strategies play a crucial role in allocating wireless channels with limited resources to appropriate IIoT devices. Use S = C N to represent the ratio of the number of IIoT devices to the number of subchannels. There are mainly three commonly used scheduling strategies: 1) Random scheduling strategy: In each round of communication, the central server randomly selects N related IIoT devices for parameter update. At the same time, a dedicated sub-channel is built between the central server and the selected user equipment to transmit training parameters.
2) Cyclic scheduling strategy: The central server divides all IIoT devices into G groups. Each time a group is selected to build a channel for communication, and the model parameters are updated cyclically during each communication. 3) Proportional fairness strategy: In each round of communication, select N from C related IIoT devices according to the following calculation method: where i = {i 1 , i 2 , ..., i N } is a vector of length N . i * = {i * i , i * 2 , ..., i * N } represents the index number of IIoT device. ρ ii,t and ρ ii,t represent the instantaneous and time average signal-to-noise ratios (SNRs) of IIoT device i n during the t-th communication, respectively.
The SNR received at the central server is expressed as follows: where β represents the link loss index, h k represents the smallscale fading, σ 2 is the variance of Gaussian additive noise, and Ψ k u represents the position of IIoT device k.
An important purpose of model training is parameter update. At the number of communication rounds t, when the central server establishes a transmission channel with a certain IIoT equipment and successfully decodes the sent data, the parameter δv t k can be updated from IIoT equipment to the central server. We use the probability of successful parameter update to indicate the transmission performance of the wireless channel, which is expressed as follows: where γ k,t represents the instantaneous and time average signal-to-noise ratio at the central server. S b k,t ∈ {0, 1} represents the selection index, where b represents different scheduling strategies. When S b k,t = 1, it means that the communication time has started between the central server and IIoT equipment k, otherwise there is no communication.

C. IIoT Equipment Node Selection Based on DRL
Due to the different uses and geographical distribution of IIoT equipments, the data they generate will have the characteristics of heterogeneity, and the quality of the data will also vary. The problems of low data training efficiency, long model aggregation time and high wireless communication cost caused by data characteristics are major challenges in the application of FL in the field of IIoT. Therefore, in order to improve the quality of model aggregation and reduce communication costs, inspired by reference [20], we use DRL algorithm to select IIoT equipment nodes with accurate learning models. The main idea is to model the local training cost of IIoT equipments and describe the problem as a MDP. We use DDPG to find the optimal solution of MDP. DRL agent continuously interacts with the environment to accumulate maximum reward and the essential purpose is to minimize the cost of FL.
We first give the cost index for selecting IIoT equipment nodes. The total cost of node selection in FL algorithm is composed of training time cost and training quality cost. The training time cost of local IIoT equipment is where In addition, we use the indicator of learning quality to characterize the loss of training accuracy: where σ i = j Loss(y j −ŷ j ) is used to quantify the quality of the network model, w t represents the training model aggregated in time slot t, and d i represents the training data of IIoT equipment i. Therefore, the total cost of DRL assisted FL algorithm in time slot t is The node selection of IIoT equipment is a combinatorial optimization problem. We model it as a MDP, denoted by M = {S, A, P A , C A }. Among them, S represents the state space of IIoT equipment node, A represents the action space, P A represents the state change probability of the action taken, and C A represents the cost of the new state produced by the action. Therefore, the selection problem of IIoT equipment nodes is expressed as: where δ t represents the selection status of the IIoT equipment node. When the IIoT equipment node i is selected, δ t i = 1, otherwise δ t i = 0. The DRL agent trains the local model by interacting with the environment, and then we use the DDPG to select the optimal solution for IIoT equipment node. In the DRL algorithm, we use the reward function to evaluate the effect of taking action a t . The evaluation method is as follows: where T m represents the number of CPU cycles required to train the model m on the data d i , µ i represents the computing resources available to IIoT equipment i, and τ represents the transmission rate of the wireless network channel. The DDPG uses a value function to determine the strategy, which mainly includes an actor deep neural network (DNN) π(s t |θ pi ) and a critic DNN Q(s t , a t |θ Q ). The DDPG uses historical experience stored in local replay memory to perform information conversion. The converted information includes the current state s t , the action taken a t , the next state s t+1 and the reward r(s t , a t ) obtained by taking the action a t . The target network generates target values to train the critic DNN model.
The specific pseudo code is given in Algorithm 1, where the client label is indexed by n, B represents the local mini batch size, E represents the number of local epoch, α represents the learning rate.

V. EXPERIMENTAL SETUP AND RESULT ANALYSIS
In this part, we evaluate the performance of FL algorithm assisted by DRL based on MNIST, Fashion MNIST and CIFAR-10 (IID and non-IID). Since the above data sets reflect the characteristics of IIoT equipment data, we mainly test the accuracy of the algorithm in training the above data sets. This shows that the FL algorithm assisted by DRL has advantages in training IIoT equipment data.

A. Experimental Setup
Considering the host performance, we mainly use several small agent data sets to test the algorithm performance. MNIST data set and Fashion MNIST data set are mainly used for number recognition tasks. The former uses a simple multilayer perception (MLP) with two hidden layers, in which each layer has 200 units and is activated by the ReLU function [38]. The latter is trained by CNN, which has a fully connected layer (512 units and ReLU function), a softmax output layer and two 5 × 5 convolution layers. To fully study FL, we use two data partitioning methods for IIoT equipments. The first is the IID of data shuffling and then divided into 100 equipment Algorithm 1 Federated learning algorithm assisted by deep reinforcement learning Input: θ π , θ Q ; 1: Set θ tar π = θ π and θ tar Q = θ Q ; 2: Initialize actor DN N and critic DN N parameters; 3: K clients, B batch size, α learning rate; 4: Server executs: 5: initialize θ 0 ; 6: for each round do 7: max(C · K, 1) → num; 8: (random clients(num)) → S t ; 9: select client n ∈ S t ; 10: ClientU pdate(n, θ t ) → θ n t+1 ;

25:
Sample a mini batch of experiences 26: f rom local replay memory; 27: update π(s|θ π ) and Q(s, a|θ Q ); 28: update the target network parameters 29: to local replay memory; 30: store the new experiences; 31: end for 32: end for 33: return θ to server; receiving 600 data samples respectively. The second is the non-IID which classifies the data according to the digital tags, which is divided into 200 pieces of 300 pieces and each client is allocated 2 pieces. This setting method can explore the breaking degree of the algorithm on highly non-IID data. CIFAR-10 data consists of 10 classes of 32 × 32 images with three GRB channels. We used 50,000 training cases and 10,000 test cases and distributed them equally to 100 equipment.

B. Performance Evaluation
We first explore the impact of the number of IIoT equipment, i.e., the number of clients, on the MNIST data set and fashion MNIST data set models. The client ratio C controls the number of multi client parallelism. Because we use 100 IIoT device as clients, when C = 0, it represents one client, followed by 10, 20, 50 and 100 clients in turn.  clients account for to reach the required precision for training the two data sets. When B = ∞ is used for data training, all 600 data samples are treated as a single batch processing in each round. At this time, increasing the proportion of clients can only show a small advantage. When B = 10 is used for batch processing (especially when C ≥ 0.1), the advantage of communication round number change is obvious. In order to balance the computational efficiency and the convergence rate, we fix C = 0.1. Figs. 3 and 4 show the gradual improvement of the accuracy of MNIST data set and Fashion MNIST data set after different training rounds. Each data set is trained based on IID data and non-IID data, respectively. It can be seen that after about 1,000 rounds of communication, the accuracy rate of MNIST data set can reach 99% and then it tends to be stable. After about 2,000 rounds of communication, the training accuracy of the Fashion MNIST data set reached 97% and stabilized.  Since the CIFAR-10 data set has no natural user partition, we consider the balancing and IID settings. The architecture of the training model is derived from Tensorflow, which mainly includes two convolution layers, two full connection layers and one linear conversion layer. In the process of training, we use stochastic gradient descent (SGD) method for small batch training with size of 100. Fig. 6 shows the relationship between the test accuracy of CIFAR-10 data set and the number of communication rounds. Through the above experiments, we can see the advantages of the FL algorithm assisted by DRL in training the data of Our algorithm 20 10 17 173 IIoT equipment. Considering the outstanding characteristics of heterogeneity and privacy of data generated by IIoT equipments, we use MNIST, Fashion MNIST and CIFAR-10 data sets to represent them. Based on MLP and CNN to train the above data sets, the training accuracy is rising in the process of continuous communication between the client and the central server. Among them, CIFAR-10 data set can achieve 85% test accuracy, while MNIST and Fashion MNIST data sets can achieve more than 98% training accuracy. Therefore, it can be shown that the FL algorithm assisted by DRL has excellent performance and has the ability to effectively manage and train the equipment data of IIoT.
In addition, we compare the algorithm proposed in this paper with a baseline algorithm called FedSGD [39]. In each round of communication, we select IIoT devices with a proportion of C, and then calculate the loss gradient of these devices. We fix the value of C to 0.1, and then discuss the number of communication rounds required for the algorithm to achieve the target accuracy when B or E changes. The results are shown in TABLE IV.
It can be seen from the table that the change of E value and B value has obvious influence on the number of communication rounds. We set the target accuracy to 98%. For IID data, increasing the value of E and reducing the value of B can significantly reduce the number of communication rounds to reach this accuracy. But for non-IID data, although the number of communication rounds will be reduced, the overall effect is not obvious. In general, the larger E is and the smaller B is, the optimization effect is more obvious. So overall, the algorithm proposed in this paper is effective in improving the data training rate. Each selected IIoT device calculates the average gradient of local data based on the current model, and then the central server aggregates these gradients and updates them using equation (4). Each IIoT device uses its local data to perform one-step gradient descent based on the current model, and then the server performs a weighted average of the generated model. The final algorithm achieved good results.

VI. CONCLUSION
IIoT is the key technology to realize industry 4.0 and it is also the objective embodiment of the development degree of industry 4.0. What cannot be ignored is that the current IIoT is facing the management and training problems brought by the explosive growth of user data. The emergence of FL provides a new solution paradigm for heterogeneous data and private data training, and it can support the development of IIoT in a way that reduces model deviation. This paper mainly aims at the problem of how to manage and train a large amount of data produced by IIoT, and proposes a FL algorithm assisted by DRL in wireless network environments. DRL based on DDPG is mainly used for the selection of IIoT equipment nodes. We fully analyze the heterogeneity and privacy of the data generated by IIoT equipments. In the experimental phase, we use MNIST, Fashion MNIST and CIFAR-10 data sets to represent IIoT equipment data. The final results show that the FL algorithm assisted by DRL can effectively train the above data sets and achieve a high accuracy rate, which shows the effectiveness of the FL algorithm assisted by DRL in the management of IIoT equipment data.


A Human-Centric Method for Generating Causal Explanations in Natural Language for Autonomous Vehicle Motion Planning

Inscrutable AI systems are difficult to trust, especially if they operate in safety-critical settings like autonomous driving. Therefore, there is a need to build transparent and queryable systems to increase trust levels. We propose a transparent, human-centric explanation generation method for autonomous vehicle motion planning and prediction based on an existing white-box system called IGP2. Our method integrates Bayesian networks with context-free generative rules and can give causal natural language explanations for the high-level driving behaviour of autonomous vehicles. Preliminary testing on simulated scenarios shows that our method captures the causes behind the actions of autonomous vehicles and generates intelligible explanations with varying complexity.

Introduction
Autonomous vehicles (AVs) are predicted to improve, among other things, traffic efficiency and transport safety, reducing road fatalities possibly by as much as 90% [Wang et al., 2022]. AVs are also predicted to decrease pollution and make transportation more accessible for passengers with disabilities. However, the current complex, highly-integrated, and often opaque systems of AVs are not easily (or at all) understood by most humans. This opaqueness often manifests in reluctance to accept the technology due to concerns that the vehicle might fail in unexpected situations [Hussain and Zeadally, 2019]. This have fostered continued distrust and scepticism of AVs in the public eye [Kim and Kelley-Baker, 2021].
We need to build trust in passengers if we want to overcome these psychological barriers and achieve wider acceptance for AVs. Crucial to the development of such trust, but neglected since the rise of black-box algorithms is the principle of explicability. This principle broadly means that the purposes, capabilities, and methods of the AV system must be transparent, that is, understandable and queryable by its passengers. While this principle is generally important for any AI system, it is especially important for AVs as they * Contact author. Figure 1: The ego vehicle (in blue) is heading straight to the blue goal but then changes lanes to the left. A passenger may inquire "Why did you change lanes instead of just driving straight?". Our system uses the motion planning and prediction calculations of the AV to give a causally justified contrastive explanation: if the ego had gone straight, then it would have likely reached its goal slower because the vehicle in front is probably changing lanes right and then exits right. operate in safety-critical settings and their decisions have farreaching consequences on human lives. There is a scientific consensus that we can increase transparency and build trust in AVs through the adoption of human-centric explainable AI (XAI) [Hussain and Zeadally, 2019;Omeiza et al., 2021b;Atakishiyev et al., 2021]. Humans prefer causal explanations [Miller, 2019], so our explanations must be causally justifiable in terms of the processes that generated our actions. We also want explanations to be intelligible for non-expert people to minimise cognitive overhead and help build general knowledge about AVs, reducing scepticism. Finally, explanations must be faithful to the decision generating process to ensure they are not misleading. We call this property the soundness of an explanation generation system.
We propose a human-centric explanation generation method called eXplainable Autonomous Vehicle Intelligence (XAVI), focusing on high-level motion planning and prediction. XAVI is based on an existing transparent, inherently interpretable motion planning and prediction system called IGP2 [Albrecht et al., 2021]. An example of the output of our system is shown in Figure 1. Our method models the cause-effect relations behind the decisions of IGP2 as a Bayesian network allowing us to draw probabilistic causal judgements about the plans of the AV. These judgements inform a context-free grammar that is used to generate intelligible contrastive explanations (see Section 2), which compare the factual observed behaviour with some counterfactual behaviour. Preliminary testing of XAVI on driving scenarios with baseline explanations by the authors of IGP2 demonstrates that our method correctly captures some of the causality behind the actions of the AV and generates intelligible natural language explanations with varying complexity. We end with a discussion outlining important future work for explaining AV behaviour. We also release the code for XAVI on GitHub 1 .

Background
Most existing methods of XAI are model-agnostic approaches that focus on classification/regression tasks relying on blackbox and surrogate models. Local surrogate models usually calculate some feature importance ordering given a single input instance for a given black box model [Ribeiro et al., 2016;Lundberg and Lee, 2017;Montavon et al., 2017]. However, instance-based explanations may ignore the overall workings of the black box and, when interpreted incorrectly, may build an incorrect understanding of our system in humans. Global surrogate models instead use black box models as a teacher to train simpler, interpretable white box systems [Bastani et al., 2017;Lakkaraju et al., 2017]. These can model the overall decision process of the teacher, though usually at the cost of soundness. In general, surrogate approaches have to introduce a new layer of abstraction that does not allow or distort the causal understanding of the decision process of the underlying black box and may introduce unwanted biases to the explanations. In addition, the output of these methods is usually expert oriented and difficult to understand for non-experts.
These shortcomings motivated several recent work that popularise a human-centric approach to explanation generation based on causality and intelligibility [Miller, 2019;Dazeley et al., 2021;Ehsan and Riedl, 2020]. In the case of classical AI planning, XAI-PLAN [Borgo et al., 2018] answers contrastive questions of the form "Why do X instead of Y?", while WHY-PLAN [Korpan and Epstein, 2018] generates natural language explanations based on model reconciliation, which compares the generated plan of the system to a usergiven alternative plan. These methods represent a shift towards a more human-centric approach, however the main issue with classical AI planning methods is their reliance on fixed domain descriptions which make their use in dynamic and complex environments such as autonomous driving difficult.
Furthermore, while the motivation for building trust and transparency for AVs is well understood, few works have proposed methods that use AV domain knowledge to inform their explanation generation system. Previous methods used deep learning to generate textual explanations for AVs based on a data set of annotated recordings with textual explanations called BDD-X [Kim et al., 2018;Ben-Younes et al., 2022]. Additionally, Omeiza et al. (2021a) proposed an explanation generation system based on decision trees taught by a black box and using language templates. These methods generate intelligible explanations, but the generating processes are surrogate models which are neither causal nor transparent.
Recently, Albrecht et al. (2021) proposed an inherently interpretable integrated planning and prediction system called 1 https://github.com/uoe-agents/xavi-ai4ad IGP2. This method relies on intuitive high-level driving actions and uses rational inverse planning to predict the trajectories of other vehicles, which are then used to inform motion planning with Monte Carlo Tree Search (MCTS). In this work, we rely on IGP2 as it is a white-box model, whose internal representations can be directly accessed while its decisions can be readily interpreted through rationality principles. Direct access to internal representations means access to the MCTS tree search which naturally allows for causal interpretation. We directly leverage this inherent causality to build our method.

IGP2: Interpretable Goal-Based Prediction and Planning for Autonomous Driving
In the following, we give a brief introduction to the notation and methods of IGP2. Let I be the set of traffic participants in the local neighbourhood of the ego vehicle denoted ε ∈ I. At time step t each traffic participant i ∈ I is in a local state s i t ∈ S i which includes its pose (position and heading), velocity, and acceleration. The joint state of all vehicles is denoted s t ∈ S = × i S i , and the state sequence (s a , . . . , s b ) is written s a:b . A trajectory is defined as a state sequence, where two adjacent states have a time step difference of one. IGP2 is goal-oriented, so it assumes that each traffic participant is trying to reach one of a finite number of possible goals g i ∈ G i . Trajectories can be used to calculate rewards r i for a vehicle, where r i is the weighted linear sum of reward components corresponding to aspects of the trajectory. The set of reward components is denoted by C and consists of time-to-reachgoal, jerk, angular acceleration, curvature, collision, and termination (received when IGP2 runs out of computational budget). Some reward components are mutually exclusive. For example, if we receive a (negative) "reward" for collision, then we cannot receive a reward for anything else.
The planning problem of IGP2 is to find an optimal policy for the ego vehicle ε that selects actions given a joint state to reach its goal g ε while optimising its reward r ε . Instead of planning over low-level controls, IGP2 defines higher-level manoeuvres with applicability and termination conditions, and (if applicable) a local trajectoryŝ i 1:n for the vehicle to follow. IGP2 uses the following manoeuvres: lane-follow, lane-change-{left,right}, turn-{left,right}, give-way, and stop. These manoeuvres are then further chained together into macro actions denoted here with ω ∈Ω, which are common sequences of manoeuvres parameterised by the macro actions. The set of all macro-actions isΩ = {Continue, Change-{left,right}, Exit, Continue-next-exit, Stop}. IGP2 searches for the optimal plan over these macro actions.
Finding the optimal plan for the ego vehicle has two major phases. First in the goal and trajectory recognition phase (referred to as goal recognition from here on), IGP2 calculates a distribution over goals G i ⊆ G i given the already observed trajectory of vehicle i denoted p(G i |s i 1:t ). To each goal we then generate a distribution over possible trajectories S i 1:n ⊆ S i 1:n given by p(S i 1:n |G i ). In the planning phase, goal recognition is used to inform a Monte Carlo Tree Search (MCTS) algorithm over macro actions, which finds the optimal sequence of macro actions (i.e. plan) by simulating many possible plans over K iterations to see how each plan interacts with the other  Step 2) From the current state (in blue) we select our next macro action based on Q-values. In this example, we selected Change-left, which is then forward simulated while the other traffic participants follow their fixed trajectories. During simulation we check for termination conditions. ( Step 3) If the ego reached a goal, or some other termination condition was met (e.g. collision), the ego receives a reward r ε which is back-propagated through the trace of macro actions that reached the termination state to update the Q-values of each action. (Step 4) We repeat the process until K iterations are reached resulting in a search tree with maximal depth dmax.

MCTS Rollouts
Figure 3: The XAVI system. IGP2 interacts with its environment and generates an optimal plan using MCTS and predictions from goal recognition. The accumulated information about how these components arrived at the optimal plan is used to build a Bayesian network (BN) model. We compare this model to a contrastive query from the passenger and extract causal relationships, which are fed to a context-free grammar that generates natural language explanations. traffic participants. More details of MCTS can be found in the caption of Figure 2. During planning we track and accumulate all relevant information about how decisions are made which we then use to initialise our explanation generation system.

eXplainable Artificial Vehicle Intelligence
Though IGP2 is transparent and can be readily interpreted, it requires expert domain-knowledge to interpret its results and present intuitive explanations that are intelligible for the non-expert passenger. It is therefore desirable to automate the interpretation and explanation procedure in the human-centric way we outlined in Section 1. We therefore present our explanation generation system called eXplainable Artificial Vehicle Intelligence (XAVI). The overall architecture of XAVI can be seen in Figure 3. The core idea of XAVI is to map the accumulated information about goal recognition and the steps of a complete MCTS planning run to random variables, which we then use to construct a Bayesian network (BN) model that encodes the properties of that particular MCTS planning run. This allows us to derive probabilistic causal information about alternative (i.e. counterfactual) sequences of macro actions and their possible effects on rewards and outcome.
Counterfactuals are a crucial part of XAVI, as the generated explanations contrast the factual, optimal plan with a counterfactual plan in which the ego would have followed a different sequence of macro actions. Contrastive explanations are studied in philosophical literature where most argue that all why-questions are (implicitly) contrastive [Miller, 2019]. This means that our generated explanations have a form similar to: "If we had done <CF> [instead of <F>], then <EFFECTS> would have happened, because <CAUSES>.". Here <F> and <CF> are the factual and counterfactual macro actions respectively, while <EFFECTS> are the changes to reward components and outcome in the counterfactual scenario. <CAUSES> describe relevant features of the trajectories of traffic participants (including the ego) that have caused the changes in <EFFECTS>. Note, we omit explicitly mentioning the factual <F> in our explanations since we assume that the passenger observed the ego's actions and is aware of what actions the ego had taken. We also assume, that the passenger's query is in a parsed format that allows us to directly extract counterfactual causal information from our Bayesian network model.

Random Variables
The first step to create the Bayesian network model is to map MCTS planning steps to random variables. MCTS starts by sampling goals and trajectories for each non-ego vehicle i. Let the vector of random variables corresponding to goal sampling (we are not sampling for the ego) be G = [G 1 , . . . , G |I|−1 ] and the vector of trajectories be S = [S 1 , . . . , S |I|−1 ]. The values of G i ∈ G i and S i ∈ S i 1:n are from the set of possible goals and trajectories for vehicle i. For example, setting G i = g i means that we sample goal g i for i.
Next, for every macro action selection step in the MCTS search tree, that is for each depth 1 ≤ d ≤ d max in the tree, we define a random variable Ω d with support ofΩ d ⊆Ω which is the set of all applicable macro actions at depth d.

Rcol R b col Ocol
Odone Figure 4: The underlying DAG of the Bayes network used for factoring the joint over all random variables. The chain-rule for Ω is not shown explicitly. The associated reward components for O done are R done ∈ {time, jerk, angular-acceleration, curvature}.
Each Ω d may also take the value of the empty set ∅, which means that no action was selected at depth d. We collect each of these random variables into a single vector denoted Ω = [Ω 1 , . . . , Ω dmax ]. This means that specifying a trace in the search tree corresponds to assigning an action to each Ω d which we can represent as a vector ω = [ω 1 , . . . , ω dmax ].
For each reward component c ∈ C that MCTS uses to calculate r ε we can define a continuous random variable R c ∈ R that gives the value for that particular reward component, or Let the vectors that collect the random variables for each component be R = [R c ] c∈C and similarly for R b .
Finally, we define outcome variables. It is important to note, that IGP2 does not explicitly represent various types of outcomes, so these variables do not correspond to any actual steps in MCTS. Instead, outcome variables are used here to conveniently describe the state of the ego vehicle at the termination of a simulation. There are four outcome types given by the set O: done (g ε was reached), collision, termination (reached d max in MCTS without reaching g ε ), and dead (for any outcomes not covered by the previous three). For each outcome type k ∈ O we define a corresponding binary variable O k ∈ {0, 1} which indicates whether that outcome was reached at the termination of a MCTS simulation. The vector of outcome variables is denoted with O.

Joint Factorisation
We now define the directed acyclic graph (DAG) to factorise the joint over random variables defined in the previous section and describe the probability distributions of each factor.
Consider the DAG shown in Figure 4. Goals for non-ego vehicles are sampled independently according to their distributions from goal recognition while the trajectories depend on the sampled goals. The joint distribution of these variables over each vehicle except the ego (hence iterating over i ∈ I \ε) are given below, which simply state that the probabilities of goals and trajectories of vehicles are mutually independent: Because trajectories of other traffic participants affect what macro actions are selected in MCTS, the random variables Ω are conditioned on S. Furthermore, the joint distribution of macro action selections Ω is given by the chain rule, which corresponds to the product of macro action selection probabilities in the MCTS tree along a search trace: The definition of p(Ω d |Ω 1:d−1 , S) corresponds to the probabilities of selecting a macro action at depth d from the unique state s reached by following Ω 1:d−1 . For each value ω d ∈ Ω d we estimate p(ω d |Ω 1:d−1 , S) from the K simulations of MCTS as the number of times ω d was selected in state s over the total number of times any action was selected in state s (i.e. the total number of visits of state s).
Reward components in R depend on the driven trajectory of the ego, and therefore the selected sequence of macro actions given by Ω. However components are otherwise calculated independently from one another. The joint distribution for R is the product of distributions of each reward component: If s now denotes the state reached by following Ω, we estimate p(R c |Ω) from the K simulations as a normal distribution with sample mean µ c (s) and sample variance σ 2 (s) calculated from the values observed for R c in state s. The existence indicator variables depend only on their corresponding reward component and their joint distribution otherwise assumes mutual independence: where p(R b c = 1|R c ) = 1 iff R c = ∅, so R b c = 1 only when we have observed some non-empty value for R c .
Finally, the outcome variables depend only on the existence of certain reward components as given in Figure 4. For each outcome variable k ∈ O let R b k ⊆ R b be the vector of random variables of binary reward components that k depends on. The joint distribution of outcomes is mutually independent: where t is not ∅. That is, the outcome k is true iff all corresponding reward components have been observed at the termination of the MCTS simulation.
Finally, by multiplying the left-hand side of Equations 1-6 we get the joint distribution over all random variables. The binary random variables R b are primarily used to simplify the calculation of the outcome probability distribution over O, so for most calculation we marginalise R b out, giving the joint we work with: p(G, S, Ω, R, O).

Note on Complexity
The size of the conditional probability distributions (CPDs) for p(ω d |Ω 1:d−1 , S) can, in the worst case, grow exponentially with the depth d according to O(|Ω| d ). However, there are two reasons why this is not a prohibitive issue. First, the search trees of IGP2 are very shallow, usually d max ≤ 4, and secondly since the MCTS tree is sparse, most values of the CPDs are zero. So instead of storing the full CPDs explicitly, we can associate each CPD to the state it is applicable in (given by Ω 1:d−1 ) and calculate the needed probabilities on-the-fly.

Extracting Causal Information
From the joint distribution we can infer various conditional distributions that allow us to draw causal judgements about counterfactual scenarios. Let us assume that MCTS selected the factual, optimal plan for the ego denoted with ω F = [ω 1 , . . . , ω dmax ]. Further assume, that the passenger query describes a (possibly incomplete) set of counterfactual macro actions ω CF = [ω j1 , . . . , ω jn ] corresponding to the random variables Ω j1 , . . . , Ω jn indexed by the set J = {j 1 , . . . , j n }.
First, we calculate the outcome distribution of O given the counterfactual, that is the distribution: This allows us to determine how the outcome of MCTS would have changed if ego had followed the counterfactual actions. Second, we want to determine how the reward components differ from the factual to the counterfactual scenario. This would allow us to order the components by the amount that they were affected by the switch to ω CF , and we can use this ordering to populate the <EFFECTS> variable in our explanation. Formally we can do this, by calculating: We can sort ∆R in decreasing order by the absolute value of its elements to get the required ordering. Finally, we would like to determine how much the trajectories of each individual non-ego participant affect the macro action selections of the ego. We can use this information to determine which traffic participants are most relevant to mention in our explanations and in what order. We can derive this ordering by comparing how the marginal distribution of macro actions p(Ω) changes when conditioned on different trajectories of non-egos. Since p(Ω) already encodes the optimal sequence of macro actions taking into account the trajectories of other participants, we are trying to find the conditional distribution that changes the marginal the least. Formally, for a vehicle i and for each of its possible goals g ∈ G i and trajectories s ∈ S i 1:n we calculate the Kullback-Leibler divergence between the marginal and the conditional of Ω: If D i g,s is the same for all goals and trajectories, that implies that the actions of vehicle i does not affect the actions of the ego, so we will ignore vehicle i. Otherwise, we can sort all D i g,s increasingly giving as an ordering on the importance of vehicles, goals, and trajectories. Note, that if vehicle i has only a single predicted goal and trajectory then D i g,s = 0. In this case we cannot use this measure to determine whether the vehicle i interacted with the ego or not. A more robust method to replace this measure would be to repeat the MCTS planning with each vehicle i removed from the simulations and looking at whether the actions of the ego have changed. This may however be computationally quite expensive to do.

Generating Natural Language Explanations
To generate intelligible explanations from the information derived in the previous section, we define a set of (recursive) generative rules given by a context-free grammar. We feed the extracted information to this grammar, which will generate a unique sentence. Since the raw generated sentences may be somewhat unnatural, we apply a post-processing step, where commonly occurring complex expressions are converted to simpler phrases (e.g. with higher time to goal → slower).
The complete set of generative rules is given in the appendix in Figure 6. To instantiate this grammar we pass the following information to it: • s: Information about the counterfactual scenario containing three fields: the counterfactual macro actions s.ω, as well as the most likely outcome s.o and its probability s.p as given by Equation 7.
• e: A list of effects on reward components of switching to the counterfactual. Each element e ∈ e contains two fields: the difference in reward e.δ as given by Equation 8, and the name of the reward component e.r corresponding to the difference.
• c: A list of causes that resulted in the effects we observed. Each cause c ∈ c has three fields: the non-ego traffic participant c.i the cause is related to, the trajectory (and the macro actions that generated it) c.ω the non-ego is taking as calculated using Equation 9, and the probability c.p of the ego taking that trajectory.
For example, assume that we give to the CFG the following:

Experiments
The criteria for human-centric AI set out in Section 1 necessitate our system to be transparent, causal, and intelligible. Our system is transparent by design, as neither IGP2 nor XAVI rely on any components that are black boxes or otherwise uninterpretable. We would then like to understand how well XAVI can capture the causal relationships when tested in realistic driving scenarios, and would also like to assess the intelligibility of our generated explanations. Considering that there is an oncoming vehicle from the right coming at high speed (vehicle 2) the action of vehicle 1 is only rational if its goal is to turn left and it is stopping to give way. Ego can use the time while vehicle 1 is stopped to turn right earlier, instead of waiting until vehicle 1 passes. The factual actions of ego is: ω 1

Generated Explanation
Continue "If ego had gone straight then it would have. . . " "likely reached the goal slower because vehicle 1 probably changes right then exits right." "likely collided with vehicle 1 because vehicle 1 probably changes right then exits right."

Exit-right
"If ego had turned right then it would have. . . " "not reached the goal." "collided with vehicle 1 because vehicle 1 probably changes right and exits right." Table 1: (Scenario S1) Explanations with one cause and one effect. Our system can successfully determine the cause and effect of the lane change and the effect of exiting right. Note, that the system also captures the possibility of collisions when the ego and vehicle 1 start close to one another, as in this case ego cannot break quickly enough to avoid vehicle 1 cutting in front of it.

Generated Explanation
Exit-straight "If ego had gone straight then it would have. . . " "not reached the goal." "not reached the goal because vehicle 1 likely turns left."

Exit-left
"If ego had turned left then it would have. . . " "not reached the goal." "not reached the goal because vehicle 1 likely turns left." Table 2: (Scenario S2) Explanations with at most one cause and one effect. Both counterfactuals result in non-completion of the ego's goal, which is correctly captured as well as the rational action of vehicle 1 turning left. Note, the vehicle described in the causes affects the motion of the ego but is not directly responsible for the counterfactual outcome of the ego not reaching its goal.

Generated Explanation
Continue "If ego had gone straight then it would have. . . " "likely reached the goal slower because vehicle 1 probably changes right then exits right." "likely reached the goal slower because vehicle 1 probably changes right then exits right and vehicle 2 exits right." "likely reached the goal slower and with more jerk because vehicle 1 probably changes right then exits right." "likely reached the goal slower and with more jerk and with less angular acceleration because vehicle 1 probably changes right then exits right." Table 3: (Scenario S1) Varying number of causes and effects for the counterfactual ω 1 CF = [Continue]. Including more information in the explanation improves its informativity, however longer explanations become more difficult to comprehend. For this, we perform a preliminary evaluation of the XAVI system in two simulated driving scenarios powered by the high-fidelity, open-source CARLA [Dosovitskiy et al., 2017] simulation environment. The scenarios used here are scenarios S1 and S2 from the evaluation section of IGP2. Albrecht et al.
give intuitive and rational explanations about the behaviour of the ego vehicle for each scenario presented in the IGP2 paper. In particular, details of scenarios S1 and S2 and the explanations of the observed behaviours are presented in Figure 5. We rely on these explanations as ground truth to see how well our generated explanations match the causal attributions of the ground truth explanations. We also vary the number of effects and causes passed to the explanation generation grammar to assess how the intelligibility of generated explanations changes with the complexity of the explanations.
To increase the diversity of our explanations, we perform ten simulations for each scenario where we randomly initialise the positions of the vehicles around their pre-defined starting points in a 10 meters longitudinal range. We also randomly initialise all vehicles' velocities in the range [5, 10] m/s. In all scenarios, the counterfactual action specifies the value for the first macro action selection random variable Ω 1 . So for example, the query "Why did you change left instead of continuing straight?" corresponds to Ω 1 = ω CF = [Continue].
For scenario S1, we test two counterfactual actions: in one the ego continues straight behind vehicle 1 until it reaches its goal, so that ω 1 CF = [Continue]; in the other, the ego turns right at the junction, so ω 1 CF = [Exit-right] 2 . For scenario S2, we test the following two counterfactual actions: ω 2 CF = [Exit-left], and ω 2 CF = [Exit-straight]. Note, that "Exit-straight" is used here to differentiate the action from regular "Continue" as the former macro action encodes giving way at a junction while the latter does not.
The generated explanations which include at most a single cause and a single effect are shown in Tables 1 and 2. Our system is able to correctly identify the effects of switching to the counterfactuals while also explaining which actions of the other vehicles (if any) caused those effects. In scenario S1, XAVI also revealed a possible collision outcome when the ego and vehicle 1 are spawned close to one another. Note, this outcome did not occur in the original IGP2 paper due to differences between random initialisations of vehicle positions.
Explanations with more than one cause or effect for scenario S1 are shown in Table 3. We do not have a similar table for scenario S2 as all relevant causal information can be captured by at most one cause. This is because the actions of vehicle 2 do not affect the actions of the ego directly in any way. For scenario S1, we can see that the shorter explanations can already capture the most crucial causal information of the ground truth, but more effects can be uncovered by XAVI. However, more detailed explanations increase the complexity of explanations which may make them harder to understand.

Discussion
Our results show that XAVI successfully captures some of the causal relationships as compared with the ground truth explanations from IGP2, while also being able to discover other, unexpected outcomes. The system is then able to generate intelligible explanations of varying complexity.
However, there are limitations to our work that need to be addressed in future work. One limitation of the method is its inability to explain the causes behind actions of traffic participants in terms of properties that are lower level than macro actions, e.g. features of raw trajectories. We would like to be able to justify our actions with causes that are finer in detail than the very high-level macro actions we currently have. High-level macro actions may encode different behaviours depending on how the other traffic participants are acting, therefore formulating causes in terms of macro actions may mask crucial differences between different runs of simulation. Indeed, the given causes for the ego's actions in Tables 1 and  3 were the same between counterfactuals, but it is clear that for different counterfactuals different causes relating to the particular motion of vehicles would be more relevant. For example, in Table 1 for the counterfactual Continue where ego reaches its goal slower, we should mention that vehicle 1 is slowing down for a turn instead of just mentioning that it is exiting right. On the other hand, for the same counterfactual where ego collides with vehicle 1 the more relevant cause for the collision is the actual fact that vehicle 1 unexpectedly changes right. To enable this lower-level extraction of causes would mean that we need to find a way to compare and filter features of trajectories based on their causal relationships to other variables, which is a difficult task given the complexity of driving environments.
Automatic explanation generation methods are by their very nature post-hoc, that is they work after our decisions were made. A common concern with any such post-hoc method is that they may not be sound, that is, faithful to the workings of the system they are explaining. Without a formal proof of soundness, we cannot fully claim that XAVI is totally faithful to IGP2. For example, XAVI does not represent each time step of the simulations explicitly or reason about how Q-values are updated. However, given the variables XAVI does reason about, we argue that our model is constructed to follow the steps of MCTS exactly without changing, removing, or adding extra information over a completed IGP2 planning run.
Another aspect to consider relates to the queries of passengers. While contrastive explanations work well for whyquestions, there are many other types of questions users may ask (e.g. "What?", "How?"), and we should support these lines of queries in the future. Parsing passengers' questions is also a non-trivial task. How could we know which macro actions a passenger is referring to in their question? What if those macro actions are not at all in our search tree? This last question also shows that we need a principled way to deal with missing data or cases where the system cannot give an explanation. What is more, giving explanations where algorithmic exceptions are present in a non-misleading and consistent way is especially important, as these explanations reveal shortcomings of our systems, that may reduce trust levels.
One aspect of human-centric AI, which we did not mention in this work is the benefit of being dialogue-oriented. Miller (2019) strongly argues that human-centric systems should be able to hold conversations with their human partners and allow opportunities for users to pose follow-up questions. This is beneficial for the users because they can converse with our system as long as their curiosity or information-need is not satisfied. Moreover, the system itself benefits from being able to hold conversations, as it can put the system on equal social status with humans, which is fundamental for developing trust [Large et al., 2017]. We may also use follow-up questions to assess the passengers' understanding of our system, and deliver relevant explanations that are specifically designed to match the individual needs of each passenger.
Our evaluation of XAVI is preliminary, though the results are encouraging. However, we need to test our system on many more interesting scenarios so that we can generate a more varied set of explanations if we want to be certain that XAVI does indeed work properly and is useful for passengers. Besides the scenarios by Albrecht et al. (2021), we can base further evaluation on the scenarios presented by Wiegand et al. (2020) which were specifically collected to evaluate XAI in autonomous driving scenarios. In the future, it will also be important to run a user study on how the generated explanations affect trust and knowledge levels in humans, as our ultimate goal is to achieve trustworthy autonomous driving. Moreover, this will help to quantitatively assess the intelligibility of generated explanations and compare XAVI to other explanation generation systems.

Conclusion
In this paper, we present an explanation generation system called eXplainable Autonomous Vehicle Intelligence (XAVI). XAVI is designed to be fully transparent, causal, and intelligible thereby building towards a more human-centric explainability approach. It is based on mapping a Monte Carlo Tree Search-based motion planning and prediction system for autonomous vehicles to a Bayesian network that models causal relationships in the planning process. Preliminary evaluation of the system on a driving scenario shows that XAVI can accurately retrieve the causes behind and the effects of an autonomous vehicle's actions, and generate intelligible explanations based on causal information. We also discuss several possible next steps and issues that need to be addressed in future work, such as lower-level causes, conversation-enabled systems, the need for error-handling, and question parsing.


extfieldsofstudy:
Computer Science
Engineering

text:
Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognition

Automatic speech recognition (ASR) systems degrade significantly in face of noisy conditions. Recently, speech enhancement (SE) has been introduced as front-end module to reduce noise and improve speech quality for ASR, but it would also suppress some important speech information, i.e., over-suppression problem. To alleviate this, we propose a dual-path style learning approach for end-to-end noise-robust automatic speech recognition (DPSL-ASR). Specifically, we first introduce clean speech feature along with the fused feature from previously proposed IFF-Net as dual-path inputs to recover the over-suppressed information. Then, we propose a style learning method to map the fused feature close to clean feature, in order to learn latent speech information from the latter, i.e., clean"speech style". Furthermore, we employ consistency loss to minimize the distance of ASR outputs in two paths to improve noise-robustness. Experimental results show that the proposed approach achieves relative word error rate (WER) reductions of 10.6% and 8.6% over the best IFF-Net baseline, on RATS Channel-A and CHiME-4 1-Channel Track datasets, respectively. Visualizations of intermediate embeddings indicate that DPSL-ASR can recover abundant over-suppressed information in enhanced speech. Our code is available at GitHub: https://github.com/YUCHEN005/DPSL-ASR.

INTRODUCTION
Automatic speech recognition (ASR) has achieved great success with the advent of deep learning techniques [1][2][3][4][5][6], which has been widely used in practical applications. However, it is still a challenging task when put under extremely noisy conditions, particularly in radio communication speech [7], which are distorted by ambient noise and communication channel due to limited transfer bandwidth.
Prior works usually introduce speech enhancement (SE) [8][9][10][11] as a front-end pre-processing module to reduce the additive noise and improve speech quality for downstream ASR task. However, recent works [12][13][14] observed that the SE processing could not always improve ASR performance, as some important information for ASR in original noisy speech are also reduced by SE, together with the additive noise. Such over-suppression problem is usually undetected during the speech enhancement stage, but could significantly degrade the performance of downstream ASR task.
To alleviate such issue, prior work [15] proposed an interactive feature fusion network (IFF-Net) with joint SE-ASR training strategy. Specifically, it designed a CNN-Attention based architecture to interactively fuse the over-suppressed enhanced speech with original noisy speech, in order to recover the lost information in enhanced speech for downstream ASR. It achieves good improvements but over-suppression problem can still be observed.
In this paper, we propose a novel dual-path style learning approach for end-to-end noise-robust automatic speech recognition (DPSL-ASR) to further alleviate over-suppression problem. Specifically, we first introduce clean speech feature along with the fused feature from IFF-Net [15] as dual-path inputs to the back-end ASR module, where the clean feature can complement the suppressed information in corresponding fused feature. We then propose a style learning method to map fused feature close to clean feature, in order to learn latent speech information from the latter, i.e., clean "speech style". Furthermore, we employ consistency loss to minimize the distance of ASR outputs in two paths to improve noise-robustness. Experimental results show that the proposed approach significantly outperforms the best IFF-Net baseline and learns more latent speech information that is over-suppressed in prior methods.
The paper is organized as follows. Section 2 introduces our proposed DPSL-ASR approach. In Section 3, experimental settings and results are described. Section 4 draws conclusions.

Overview
In this work, we propose a novel dual-path style learning system for end-to-end noise-robust automatic speech recognition (DPSL-ASR), which is illustrated in Figure 1 (c).
We first examine a joint training system [9,11] in Figure 1(a) by cascading the front-end SE module and back-end ASR module via multi-task learning strategy. However, the over-suppressed speech generated by speech enhancement module could significantly degrade the performance of downstream ASR task. To alleviate this, recent study [15] proposed an IFF-Net to combine the enhanced Fbank feature XE and original noisy Fbank feature XN as a fused feature XF to recover the over-suppressed information for downstream ASR task, as shown in Figure 1(b). The IFF-Net has achieved significant improvement of ASR performance, but over-suppression phenomenon can still be observed in its generated fused feature XF .
To further alleviate the over-suppression problem, we first introduce clean speech feature XC along with the fused feature XF from IFF-Net [15] as dual-path inputs to the back-end ASR module, as shown in Figure 1(c). Such parallel clean feature can provide more complementary information that has been suppressed in fused feature. In addition, it can also benefit the ASR training, especially at the early training stage when the poorly-trained speech enhancement module and IFF-Net cannot provide high-quality fused feature XF for ASR. Then, we propose a novel style learning method to map fused feature close to the clean feature, in order to learn latent speech  information from the latter, i.e., clean "speech style". Furthermore, we employ consistency loss to minimize the distance of ASR outputs in two paths to improve noise-robustness. As a result, the back-end ASR module could learn more noise-robust speech representations to alleviate the over-suppression problem, and thus yield better performance. Since no clean data in test set is available, the clean path is discarded during inference stage.

Style Learning
Inspired by the image style transfer algorithm in prior work [16], we propose a style learning method to transfer "speech style" from clean encoded embeddings to fused embeddings, as shown in Figure 2. Specifically, we first send clean feature and fused feature XC , XF ∈ R T ×F into the ASR module in two paths, where T is number of time frames and F is number of frequency-bins. The ASR module in two paths share parameters with each other, which consists of L Conformer layers in the encoder and L/2 Transformer layers in the decoder, following the prior work [17]. Based on this, we denote the clean embedding from l-th encoder layer as E l C and the fused embedding from l-th encoder layer as E l F , where E l C , E l F ∈ R T ×D , l ∈ {1, ..., L} and D is embedding size. The "speech style" of clean embedding E l C and fused embedding E l F from l-th encoder layer are then formulated as: Equation 1 conducts dot product between the encoded embeddings and their own transpose, generating the style matrixes S l C , S l F ∈ R D×D that indicate correlations between every two embedding channels. As a result, the style matrixes contain abundant latent information inside the speech feature, which is thus defined as "speech style". Then we formulate the style loss LossSL as follows: where · 2 denotes L2 norm and D 2 in the denominator is used for normalization. Without specified, we employ the embeddings from all L encoder layers to calculate the style loss. In this way, the ASR encoder would be optimized by style loss to learn abundant latent speech information from the clean embeddings, which could recover the suppressed information in fused embeddings and thus alleviate the over-suppression problem in IFF-Net.

Consistency Loss
The consistency loss is designed to minimize the distance of outputs in two paths after ASR decoder via the symmetric Kullback-Leibler (KL) divergence [18,19], which is formulated as follows: where DC , DF ∈ R U ×V are posterior outputs of ASR decoder (before softmax), in clean path and fused path respectively. U is the number of output tokens and V is the vocabulary size. We aim to minimize the KL-divergence-based consistency loss to push the decoded embedding in fused path closer to that in clean path. As a result, the ASR output could be more robust against the noisy and over-suppression conditions.

Training Strategy
The proposed DPSL-ASR system adopts the following optimization strategy during training process: • Seeking parameters of the SE module to minimize the mean square error loss of the speech enhancement task Loss Enh ; • Seeking parameters of the IFF-Net and ASR module to minimize the ASR loss Loss C ASR for clean path and Loss F ASR for fused path; • Seeking parameters of the ASR module to simultaneously minimize the style loss LossSL for ASR encoder and the consistency loss LossCL for ASR decoder.
The entire DPSL-ASR system is trained in an end-to-end manner, where the overall training objective Loss f inal is formulated as: where λASR, λSL, λCL and λ f used are weighting parameters to balance different training objectives.

Datasets
We conduct experiments on two corpora: one is Robust Automatic Transcription of Speech (RATS) [20], which is extremely noisy radio communication data; another is the far-field CHiME-4 [21] dataset that contains normally noisy speech.
RATS dataset consists of eight parallel channels and we only use the Channel-A subset in this work, which contains 44-hour training data, 5-hour valid data and 8-hour test data. As RATS dataset is chargeable by LDC, we release its fbank features and several listening samples on GitHub for reference 1 .
The CHiME-4 dataset 2 consists of three subsets: clean data, real noisy data and simulated noisy data. The clean data comes from the WSJ0 [22] training data. The real noisy data is recorded in four noisy environments, including bus, cafe, pedestrian area and street junction. The simulated noisy data is generated by mixing the clean data with background noise recorded in above four environments. We use real and simulated noisy data of 1-channel track in this work.

Network Configurations
The proposed DPSL-ASR system consists of three modules: the SE module, the IFF-Net and the ASR module. The SE module is same as that in [15], which utilizes 3 layer of 896-unit bi-directional long short-term memory (BLSTM) [23], followed by a 257-unit linear layer and a ReLU [24] activation function to predict masks for noisy magnitude features. The IFF-Net consists of 4 residual attention blocks with 64 filters, following the best configurations in prior work [15]. The ASR module contains 12 Conformer [17] layers in encoder, and 6 Transformer [3] layers in decoder. The embedding dimension/feed-forward dimension/attention heads are set to 256/2048/4 for all the Conformer and Transformer layers. We also employ 1000 byte-pair-encoding (BPE) [25] tokens as ASR output.
The network is optimized by Adam algorithm [26], where the learning rate first warms up linearly to 0.002 in 25,000 steps and then decreases proportional to the inverse square root of the step number. The batch size is set to 64. The training epoch is set to 50 for experiments on RATS dataset and 100 for experiments on CHiME-4 dataset. The weighting parameters λASR, λSL, λCL and λ f used are set to 0.7, 0.01, 0.4 and 0.3 respectively, where we first tune λ f used to build LossASR, and then tune λASR to combine Loss Enh , followed by λSL to add LossSL, and λCL to add LossCL. We also build a 2-layer 650-unit RNNLM on training text for rescoring during inference. All hyper-parameters are tuned on validation set.

Reference Baselines
We implement four competitive baselines to evaluate our proposed DPSL-ASR approach: • E2E-ASR [17]: a Conformer-based end-to-end ASR system.  Table 1. WER% results of the proposed DPSL-ASR and four baselines on RATS Channel-A dataset. "Use SL & CL" denotes whether use the style learning and consistency loss.

Method
Use SL & CL WER(%) E2E-ASR [17] 54.3 Cascaded SE-ASR [8] 53.1 Joint SE-ASR [9] 51.8 IFF-Net [15] 46.2 DPSL-ASR (ours) 44.1 41.3 Table 2. WER% results of the proposed DPSL-ASR and four baselines on CHiME-4 1-Channel Track dataset. "Use SL & CL" denotes whether use the style learning and consistency loss. "Dev" and "Test" denote WER% results on development set and test set, respectively. "real" and "simu" denote the real noisy subset and simulated noisy subset, respectively. • Cascaded SE-ASR [8]: a cascaded SE and ASR system that only uses final ASR loss for optimization. • Joint SE-ASR [9]: a cascaded SE and ASR system that employs multi-task learning strategy for joint SE-ASR training, as shown in Figure 1(a). • IFF-Net [15]: a joint SE-ASR system with IFF-Net, as shown in Figure 1(b). The IFF-Net fuses enhanced speech and original noisy speech to learn a fused representation for ASR. Therefore, the ASR module could learn complementary information to alleviate over-suppression problem. For fair comparison, the SE modules, IFF-Nets and ASR modules in all baselines are in same structures and configurations with the proposed DPSL-ASR. Therefore, our DPSL-ASR contains same number of parameters as the IFF-Net baseline.

Results
We report experimental results in terms of word error rate (WER), as our target is ASR performance while SE is only auxiliary task. Table 1 summarizes the comparison between the proposed DPSL-ASR and the four baselines on RATS Channel-A dataset. We observe that the E2E-ASR system achieves 54.3% WER result, indicating the high difficulty of recognizing speech under extremely noisy conditions. Compared with E2E-ASR system, the cascaded SE-ASR approach reduces 1.2% WER absolutely by introducing speech enhancement as the front-end module to reduce noise for downstream ASR task. Based on this, the joint SE-ASR approach obtains another 1.3% absolute WER reduction by optimizing the SE and ASR modules simultaneously with multi-task learning. Furthermore, the IFF-Net baseline improves significantly with 5.6% absolute WER reduction (51.8%→46.2%), by interactively fusing the enhanced speech Table 3. WER% results of the style learning and consistency loss with the proposed DPSL-ASR system on RATS Channel-A dataset. "Use SL" denotes whether use style learning, "Use CL" denotes whether use consistency loss. The weighting parameters λSL and λCL are kept same as those described in Section 3.2.1.

Method
Use  Table 4. WER% results of style learning on different encoder layers, with RATS Channel-A dataset. "0" denotes do not use style learning, "1-3" denotes use it on first 3 layers, "1-12" denotes use it on all 12 layers, etc. The style losses on selected encoder layers are averaged like Equation 2, and the weighting parameter is same as before.

Method Encoder Layers WER(%)
DPSL-ASR and original noisy speech to complement the lost information caused by over-suppression. Finally, we observe that our proposed DPSL-ASR obtains the best result with 10.6% relative WER reduction over the best IFF-Net baseline (46.2%→41.3%). Table 2 further compares the proposed DPSL-ASR with reference baselines on CHiME-4 1-Channel Track dataset. We observe that our DPSL-ASR approach obtains average relative WER reduction of 8.6% over the best IFF-Net baseline, which verifies its effectiveness under normally noisy conditions. Therefore, we conclude that the proposed DPSL-ASR achieves superior performance on both extremely noisy radio-channel RATS data and the normally noisy far-field CHiME-4 data.

Effect of Style Learning and Consistency Loss
We further report the effect of proposed style learning and consistency loss on the performance of DPSL-ASR system, as presented in Table 3. Firstly we build the DPSL-ASR system with dual-path inputs, which achieves a WER of 44.1% (vs. 46.2% in IFF-Net baseline). Then, we observe that using style learning method to learn clean "speech style" can improve the WER result by 1.7% absolutely (44.1%→42.4%). Besides, applying consistency loss on decoded embeddings can also improve the performance (44.1%→43.3%). Furthermore, we can achieve the best result with 2.8% absolute WER improvement (44.1%→41.3%) by using both of them. Therefore, we conclude that both of the proposed style learning and consistency loss contribute positively to the superior performance of DPSL-ASR, where style learning plays the dominant role.

Effect of Style Learning on Different Encoder Layers
To understand the principles of style learning, we report the WER results of applying it on different encoder layers in Table 4. We observe that the style learning on low encoder layers can improve some WER performance (44.1%→43.9%/43.4%), while applying it on high encoder layers achieves significantly more improvements (44.1%→43.3%/42.7%). The reason could be that the low-layer en- coded embeddings focus more on speech features with phonetic information, while high-layer embeddings contain richer semantic representations with linguistic information that is closer related to ASR task, so that more valuable clean "speech style" could be learned on high encoder layers. The best result is achieved by performing style learning on all 12 encoder layers (44.1%→42.4%). Therefore, we conclude that both low-layer and high-layer style learning can improve the ASR performance, where the latter is more effective.

Visualizations of ASR Intermediate Embeddings
To further analyze the contribution of proposed style learning and consistency loss, we visualize the ASR intermediate embeddings in DPSL-ASR with a RATS Channel-A test sample (fe 03 14342-04710-A-024073-025522-A.wav), as shown in Figure 3. We first present the encoded embeddings after final ASR encoder layer with and without style learning in (a-c). We observe that the encoded embedding (a) in fused path loses many latent speech information compared to (c) in clean path, where figure (a) looks blurred while (c) contains clear textures. After applying style learning to map encoded embeddings from fused path to clean path, the fused embedding (b) learns abundant clean "speech style" to alleviate over-suppression, i.e., it shows clearer textures than (a). Furthermore, we visualize the decoded embeddings after final ASR decoder layer with and without consistency loss in (d-f). We first observe clear difference between the decoded embedding (d) in fused path and (f) in clean path, where figure (d) looks bright but (f) looks dark. After applying the consistency loss, the fused embedding (e) is pushed much closer to the clean path, i.e., it looks darker than (d) in figure.

CONCLUSION
In this paper, we propose a dual-path style learning approach for end-to-end noise-robust automatic speech recognition (DPSL-ASR) to alleviate the over-suppression problem. Specifically, we first introduce clean speech feature along with the fused feature from IFF-Net [15] as dual-path inputs to recover the suppressed information. Then, we propose a style learning method to map the fused feature close to clean feature, in order to learn abundant latent speech information from the latter, i.e., clean "speech style". Furthermore, we employ consistency loss to minimize the distance of ASR outputs in two paths to improve noise-robustness. Experimental results on RATS Channel-A and CHiME-4 1-Channel Track datasets show that the proposed DPSL-ASR approach effectively alleviates the oversuppression and significantly outperforms the competitive baselines.