---
abstract: 'With the development of communication technologies, connected autonomous vehicles (CAVs) can share information with each other. Besides basic safety messages, they can also share their future plan. We propose a behavior planning method for CAVs to decide whether to change lane or keep lane based on the information received from neighbors and a policy learned by deep reinforcement learning (DRL). Our state design based on shared information is scalable to the number of vehicles. The proposed feedback deep Q-learning algorithms integrate the policy learning process with a continuous state space controller, which in turn gives feedback about actions and rewards to the learning process. We design both centralized and distributed DRL algorithms. In experiments, our behavior planning method can help increase traffic flow and driving comfort compared with a traditional rule-based control method. It also shows the distributed learning result is comparable to the centralized learning result, which reveals the possibility of improving the policy of behavior planning online. We also validate our algorithm in a more complicated scenario where there are two road closures on a freeway.'
author:
- Songyang Han
- 'Fei Miao [^1]'
title: |
    **Behavior Planning For Connected Autonomous Vehicles Using\
    Feedback Deep Reinforcement Learning**
---

Autonomous vehicle, behavior planning, reinforcement learning, information sharing.

Introduction
============

The development of the Dedicated Short-Range Communication (DSRC) technology [@tangade2018decentralized] and 5G technology enables Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication. The U.S. Department of Transportation (DOT) estimated that DSRC based V2V information sharing could potentially address up to 82% of crashes in the United States and prevent thousands of automobile crashes every year [@DSRC_standard]. When Basic Safety Message (BSM), including current velocities and positions, is communicated for Connected Autonomous Vehicles’ (CAVs) coordination, control approaches for scenarios such as cross intersections or lane-merging [@Coordinate_CAV; @CV_intersection; @ort2018autonomous] have shown the advantage of information sharing.

When a lane-changing decision has already been made, continuous state space controllers are designed for vehicles to merge into or leave the platoon [@li2017optimal]. However, existing control frameworks for CAVs, such as platooning, Adaptive Cruise Control (ACC) [@correct_ACC], and Cooperative Adaptive Cruise Control (CACC) [@cacc_platoon] algorithms mainly focus on the controller design when a decision about lane changing or keeping has been provided [@Survey_CVcontrol]. Methods designed based on Model Predictive Control (MPC) method [@gritschneder2018fast] or hybrid system model usually aims to guarantee string stability of the platoon or improves fuel efficiency [@savefuel_Karl]. Work in [@PredictPolicyLearn_LeCun] uses vision to predict trajectory and speed for reinforcement learning training, but it seems from the video the cars do not change lane.

Reinforcement learning (RL) has shown great success in robotics research [@kober2013reinforcement]. For solving autonomous vehicle’s challenges, one paramount concern raised by RL is how to ensure the safety of the learned driving policy. The authors in [@shalev2016safe] have proved that it cannot ensure driving safety by merely assigning a negative reward for accident trajectories, because it enlarges the variance of the expected reward, which in turn prohibits the convergence of stochastic gradient descent algorithm for policy learning.

It is efficient if autonomous vehicles can learn a driving policy to output control signals for steering angle and acceleration directly based on observed environments, such as an end-to-end convolutional neural network with front camera images as input [@codevilla2018end]. However, it only considers lane-keeping scenarios without lane-changing cases, and this pure learning method cannot provide a safety guarantee. In order to add safety guarantees, researchers start to use deep learning to learn a policy that has similar control output with MPC. For example, paper [@chen2018approximating] uses a constrained neural network to approximate MPC law. The guided policy search in [@MPCGPS_icra16] uses MPC to generate samples to train Deep Reinforcement Learning (DRL), where DRL and MPC have the same action/decision space. Though it is an approximation of MPC, the learned policy does have the same safety property with the MPC.

The other popular approach is to decompose the learning and control phases. Learning-based methods can make a high-level decision, including lane-changing or lane-keeping. For example, the work in [@pan2017virtual] evaluates the effectiveness of making high-level decisions, such as “go straight” and “go left”, based on images. The work discussing safety in [@shalev2016safe] also separates learning and control phases: first learn a policy to determine the target speed, lateral position, whether to take away or give way to the other vehicle, then conduct trajectory planning with hard constraints to ensure safety.

Certainly, splitting learning and control steps is a good way to combine the advantages of both of them. Is there a better way to integrate them? Can control help the learning process? Also, when an autonomous vehicle gets extra knowledge about the environment via V2V or V2X communication, how to make tactical decisions such as whether to change lane or keep lane to improve traffic efficiency, whether sharing future planned velocity or trajectory can bring benefits are still unsolved challenges. The existing literature of multi-agent reinforcement learning [@zhang2019multi] has not fully solved the CAV challenges yet, since either how communication among agents will improve systems performance or policy learning is not analyzed clearly.

Hence, in this work, we design a learning algorithm that uses shared information to enhance operational performance for future CAVs, considers feedback from controllers and safety requirements, and shows the benefit of V2V communication. We focus on the behavior planning challenge to determine when to change/keep lane for connected autonomous vehicles. We assume that autonomous vehicles can share current and future plans with their neighbors. We model the problem as a Markov decision process and solve it by improving the deep Q-learning algorithm for autonomous vehicles, where the behavior action a vehicle can take and the reward from this action also use the feedback control decisions of the continuous state space controller of this vehicle. In experiments, we show that feedback DRL based behavior planning can help increase traffic flow and provide a more comfortable driving experience.

The main contributions of this work are:

-   The state space design of the RL approach for CAV reduces the curse of dimensionality. The approach is scalable to the number of vehicles since the dimension of the state space is not directly relevant to the number of vehicles that share information.

-   The feedback deep Q-learning proposed in our work integrates the strength of both the reinforcement learning and optimal control. The learning algorithm can explore historical data to find a good policy under a complex environment. The continuous state space controller has a well-studied property to consider safety constraints. Controller gets involved in the proposed learning process by providing feedback to actions and rewards.

-   We design a distributed policy learning algorithm that does not rely on a cloud center to collect data or run the training process. The experiment results reveal the potential to apply both centralized and distributed learning algorithms for behavior planning.

-   The case study shows that the behavior planning policy learned by the proposed approach can deal with complicated scenarios such as road closures.

The rest of this paper is organized as follows. In Section \[sec:problem\], the behavior planning problem is described. Also, the information sharing assumption and a physical model are introduced for CAVs. In Section \[sec:learning\], we define the behavior planning as a Markov decision process and solve it through feedback deep Q-learning with both centralized and distributed versions. Experiment results are shown in Section \[sec:experiment\] concerning traffic flow and driving comfort. Conclusions are given in Section \[sec:conclusion\].

Problem Description {#sec:problem}
===================

The V2V and V2I communications extend the information gathered by a single-vehicle further beyond its sensing system. This work addresses how to utilize information sharing to make better decisions in terms of when to change/keep lane for Connected Autonomous Vehicles (CAVs). This section introduces information sharing considered in this work. Afterward, the kinematic bicycle model is briefly introduced for the continuous dynamics of an ego vehicle.

![The concept diagram of behavior planning.[]{data-label="fig:behavior_diagram"}](behavior_diagram.pdf){width="2.5in"}

As shown in the concept diagram in Fig. \[fig:behavior\_diagram\], the problem considered in this paper is to study a decision making algorithm for behavior planning, assuming that information sharing is capable between CAVs. Each vehicle is expected to share a sequence of future velocities and lane numbers with its neighbors. A formal definition of the neighbors is given in the subsequent section. Once a decision is made, the action is implemented by a controller. Because the action given by the RL agent might not be safe and feasible, a controller can change the action to guarantee safety.

Meanwhile, this action provides feedback to the RL process, which modifies the traditional learning framework as introduced later in Section \[sec:centralized\_learning\]. The corresponding controller is called a feedback controller. There are several state-of-art controllers proposed by researchers, such as Model Predictive Control (MPC) [@cesari2017scenario; @Survey_CVcontrol], and the control barrier function based controller [@chen2017obstacle; @ames2017control].

Information Sharing
-------------------

Autonomous vehicles are assumed to share information with their $\epsilon$-neighbors defined as follows.

One vehicle $j$ is said to be the $\epsilon$-neighbor of a vehicle $i$ if $\mid x^i_\tau- x^j_\tau \mid \leq \epsilon$, where $i$ and $j$ are vehicles’ indexes, $x^i_\tau$ and $x^j_\tau$ represent the longitudinal positions of vehicle $i$ and $j$ at time instant $\tau$ respectively, and $\epsilon$ is a constant parameter. The set that includes all the $\epsilon$-neighbors of the vehicle $i$ except for $i$ itself is denoted as $\mathcal{N}_i (\epsilon)$. This set is called vehicle $i$’s $\epsilon$-neighbors.

In this work, we consider autonomous vehicles driving on a 3-lane freeway. Each autonomous vehicle is assumed to share its current state and future plan, denoted by a sequence of $[v_\tau$, $v_{\tau+1}$, ..., $v_{\tau + T}]$ (velocities); $[l_\tau$, $l_{\tau+1}$, ..., $l_{\tau + T}]$ (lane numbers, labeled as 1, 2, 3) with its $\epsilon$-neighbors. Note that, though a lane number can be replaced by the lateral position of a vehicle, it can reflect lane-changing actions more clearly. According to the current development of BSM and DSRC security, the message can be both authenticated and encrypted [@tangade2018decentralized]. Hence, in this work, we assume that vehicular communication is true information that is not manipulated by attackers. Parameter $T$ determines how much future information is shared. There is a trade-off for selecting $T$. The larger $T$ means there is more information shared among autonomous vehicles which may bring more benefits. However, the communication cost would increase and the information reliability may also decrease accordingly. There exist some experiments quantifying the communication delay using WiFi, 4G and 3G network [@tangade2018decentralized]. Limited to the scope of this work, we assume that $\epsilon$ and $T$ are given based on the communication capability, and how to select a better $\epsilon$ and $T$ is one direction for future work.

Physical Model
--------------

![Kinematic bicycle model attached to the lane centerline.[]{data-label="fig:BicycleModel"}](BicycleModel2.pdf){width="2.6in"}

The physical dynamics of an ego vehicle is described by a kinematic bicycle model, as shown in Fig. \[fig:BicycleModel\]. This model can achieve the balance between accuracy and complexity [@brito2019model; @kang2018multirate]. Point $A$ represents the two left and right front wheels, while the rear two wheels are represented by point $B$. Point $O$ is the Center of Gravity (CoG). The lengths of the line segments $OA$ and $OB$ are represented by $l_f$ and $l_r$, respectively. The $\delta$ is the steering angle for the front wheels. The planar motion of this vehicle is described by three coordinates: $x$, $y$, and $\psi$. $(x,y)$ is the location of the CoG, and $\psi$ illustrates the orientation of the vehicle. The $X$-axis represents the lane centerline. The $v$ is the velocity at the CoG, and the slip angle $\beta$ denotes its angle with $OA$. The discrete-time equations of this model can be obtained by applying an explicit Euler method with a sampling time $\tau_{cs}$ for continuous states [@cesari2017scenario]. The control vector for this vehicle is defined as $ \textbf{u}_\tau \triangleq [ \delta_\tau, a_\tau ]$, where $a_\tau$ is its acceleration. The continuous state vector is defined as $\textbf{s}_\tau \triangleq[x_\tau,y_\tau,\psi_\tau,v_\tau,l_\tau]$, where $l_\tau$ is the current lane number. The detail equations can be found in the Appendix. More compactly, the update of the state vector is denoted as $\textbf{s}_{\tau+1} = f( \textbf{s}_\tau , \textbf{u}_\tau)$.

Behavior Policy Learning {#sec:learning}
========================

The behavior planning problem is modeled as a Markov Decision Process (MDP). This section first introduces the action space, state space, and reward function of this MDP. Then, a feedback deep reinforcement learning algorithm (centralized version) is proposed to solve it. This algorithm can improve the traffic flow and driving comfort of autonomous vehicles, but its learning process relies on a cloud center. Therefore, a distributed version is put forward afterward to get rid of this dependency.

![The scenario of a 3-lane freeway. The ego vehicle’s behavior is determined based on four states in the boxes.[]{data-label="fig:freeway"}](highway.pdf){width="3in"}

Action Space
------------

In this work, we consider a scenario where autonomous vehicles running on a 3-lane free freeway, as shown in Fig. \[fig:freeway\]. In behavior planning, the action space $A_t$ considers:

-   Keep Lane (KL),

-   Change Left (CL),

-   Change Right (CR).

More complicated actions are either included in these actions or represented by their combinations. For example, an overtake is the combination of change left followed by change right.

### Keep Lane

Vehicles stay in the current lane with this action. A feedback controller will adjust the speed of an ego vehicle according to the safety interval requirements, which is introduced in Sec. \[sec:continuous\_control\]. The ego vehicle may either accelerate or decelerate according to the control output of the feedback controller. In extreme cases, when the headway is too tight, the ego vehicle will fully stop on the current lane.

### Change Left & Change Right

With these two actions, an ego vehicle changes to a neighbor lane on its left/right. CL is symmetric to CR. After receiving the CL/CR action from a DRL agent, a feedback controller will implement this action based on the current traffic.

State Space {#sec:state}
-----------

The incentive to change lane is to achieve higher speed or to avoid obstacles/traffic jams. Intuitively, the states can be designed to include the positions, velocities, and lane numbers of all the vehicles. However, the number of states will increase proportionally to the number of vehicles, and the computational complexity of behavior planning will increase exponentially. To reduce the curse of dimensionality, two state functions are defined: $S_v(t, l_k)$ is used to evaluate the future velocity quality of lane \# $l_k$ based on shared information; $S_f(t)$ is used to record the lane-changing frequency of each vehicle. As an example shown in Fig. \[fig:freeway\], the state space includes:

-   $S_v(t, l_k - 1)$ for the left lane

-   $S_v(t, l_k)$ for the current lane

-   $S_v(t, l_k + 1)$ for the right lane

-   $S_f(t)$ for the lane-changing frequency

The state function for future velocity of lane \# $l_k$ is $$S_v(t, l_k) = \frac{1}{N}\sum_{i :  i \in \mathcal{N}_k } \sum^T_{j=1} \gamma^{(j-1)} v^{i}_{\tau(t)+j},
 \label{equ:state_velocity}$$ where $\mathcal{N}_k = \{i | i \in \mathcal{N}_i (\epsilon), l_t^i = l_k\}$, $N= \mid \mathcal{N}_k  \mid$, $l_k \in \{1,2,3\}$, $\gamma$ is a decay coefficient, $v^{i}_{\tau(t)+j}$ is the velocity of the vehicle $i$ at time instant $\tau(t)+j$.

This state function is the average discounted future velocity of vehicles on lane \# $l_k$. It can reflect the potential velocity quality of lane \# $l_k$. It also helps to avoid unnecessary lane-changing. For example, even though at time $t$ the vehicles on the neighbor lane have a higher speed than the ego vehicle, there may be a traffic jam in front of them. The ego vehicle could not observe this traffic jam because it could not see through its neighbors, which block its view. In this case, there is no need to change lane. The farther the time is, the less accuracy $v^i_{\tau(t)+j}$ will have. Therefore, a decay coefficient $\gamma < 1$ is multiplied to penalize future information.

The state function needs support from information sharing of both future velocities and lane numbers. The lane numbers are used to allocate which vehicle is on lane \# $l_k$, as the state function is defined for a specific lane. When the ego vehicle is on lane \# 1 or \# 3, the left/right lane’s state is set to be 0. For example, $S_v(t, 0) = 0$.

The state function for future velocity is scalable according to the number of vehicles. Unlike the aggregation of all vehicles’ states, the average discounted future velocity always exist no matter how many vehicles there are. Note that when there is no vehicle nearby, the future velocities are intentionally set to be $v^{max}$. Also, it is robust to potential package drops. Similarly, though the package drops may occur, the average velocity will not be affected dramatically because of losing one or two data points.

The state function for lane-changing frequency is $$S_f(t) = - \sum^{T_f}_{i=1} I_{change}(t-i),$$ where $t$ is the current time instant, $T_f$ is a constant determining the window size of $[t-T_f, t-1]$ and the lane changing indicator $$I_{change}(i) = \begin{cases} 
             1, & \text{if lane changing starts from the time i}; \\
            0, & \text{otherwise}. \end{cases}$$

Passengers may feel uncomfortable if a vehicle changes lane frequently. This frequent behavior will result in a smaller state function value, so there is a minus sign in front.

Reward Function
---------------

A reward is assigned to each state-action pair. For connected autonomous vehicles, behavior planning in our work targets to improve two system-level evaluation criteria: traffic flow $\mathcal{F}$ and driving comfort $\mathcal{C}$. When there is a central cloud center to collect information from all the agents, a global reward function can be used for multi-agent RL [@wai2018multi]. Therefore, the reward function is defined as a weighted sum of the above two criteria:

The global reward function (for centralized DRL) is $$R(s_v, s_f, a) = w \cdot \mathcal{F} +  \mathcal{C},
 \label{equ:reward_centralized}$$ where $s_v \in S_v$, $s_f \in S_f$, $a \in A_t$, $w$ is a trade-off weight.

### Traffic Flow

Traffic flow reflects the quality of the road throughout with respect to traffic density. Traffic density $\rho$ is the ratio between the total number of vehicles and road length. Traffic flow is calculated as $$\mathcal{F} = \rho \times \bar{v},$$ where $\bar{v}$ is the average velocity of all the vehicles [@rios2018impact].

### Driving Comfort

The driving comfort of a road segment is defined to be the average driving comfort of all the vehicles on this segment. The driving comfort of a single-vehicle is related to its acceleration and driving behavior. Define the driving comfort for the vehicle $i$’s acceleration $a_{it}$ at time $t$ and its action $ a \in A_t$ as follows: $$\mathcal{C}_{single}(a_{it}, a) = \begin{cases} 
             3, & \text{if } a_{it}< \Theta_a \text{and } a = KL; \\
            2, & \text{if } a_{it} \geq \Theta_a \text{and } a = KL; \\
             1, & \text{if } a = CL/CR, \end{cases}$$ where $\Theta_a$ is a predefined acceleration threshold.

Therefore, driving comfort (for the freeway) is calculated as $$\mathcal{C} = \frac{1}{N \cdot T} \sum_{i}\sum_t \mathcal{C}_{single}(a_{it}, a),$$ where $N$ is the total number of all the vehicles.

Feedback Deep Q-Learning {#sec:centralized_learning}
------------------------

The environment model is not available for behavior planning, so policy/value iteration does not work. Traditional sample-based reinforcement learning is good at exploiting historical data to find a good policy. However, it does not guarantee either the optimality or the safety property of the converged policy. Different from traditional Deep Reinforcement Learning (DRL), a feedback controller will monitor the action given by the DRL agent and guarantee safety.

![The agent-environment interaction in the feedback deep reinforcement learning.[]{data-label="fig:interaction"}](interaction.pdf){width="3in"}

Agent-environment interaction is shown in Fig. \[fig:interaction\] for the feedback DRL. The DRL agent is like a smart toddler who is good at learning new knowledge, but she may do something dangerous, such as eating inedible things and playing sharp items. In this case, an adult (the feedback controller) needs to look after her and guide her as a guardian. For the ego vehicle, Change Left (CL) or CR may be unsafe under some traffic scenarios. The controller will monitor action $A_t$ from the DRL agent and determine feedback action $A'_t$ based on the current traffic condition. In traditional DRL, transition experience is represented by $(s, a, r, s')$, where $s$ is the current state, $a$ is the action taken by the DRL agent, $r$ is the resulted reward, and $s'$ is the next state. In feedback deep Q-learning, transition experience is represented by $(s, a', r, s')$, where $a'$ is the feedback action executed by the feedback controller. In feedback deep Q-learning (centralized version), every vehicle will contribute its transition experience to a shared replay memory. This algorithm is shown in Alg. \[alg:centralized-qlearning\].

Initialize replay memory $D$ with a capacity $N$ Initialize action-value function $Q$ with random weights $\theta$ Initialize target action-value function $\hat{Q}$ with weights $\acute{\theta} = \theta$

Feedback Controller {#sec:continuous_control}
-------------------

The MPC controller proposed in [@cesari2017scenario] is used as an example to show how a feedback controller works. For each $T$ time horizon $\mathcal{T}=[\tau,\tau+1, ..., \tau+T-1]$, the control inputs can be generated by the following optimization program: $$\begin{aligned}
\begin{split}
&\min \limits_{\textbf{u}_k, \textbf{s}_{k+1}} \quad \sum_{k \in \mathcal{T}} (\textbf{s}_{k+1}-\textbf{s}^{ref}_{k+1})^{\mathsf{T}} Q(\textbf{s}_{k+1}-\textbf{s}^{ref}_{k+1})+\textbf{u}^\mathsf{T}_k R\textbf{u}_k \\ & \begin{array}{r@{\quad}c@{}c@{\quad}c}
s.t.&\textbf{s}_{k+1} = f(\textbf{s}_{k},\textbf{u}_{k}),   &\text{  }\forall k \in \mathcal{T}\\
 &\textbf{u}^{min} \leq \textbf{u}_k \leq \textbf{u}^{max} ,  &\text{  }\forall k \in \mathcal{T} \\
 &\dot{\textbf{u}}^{min} \leq \textbf{u}_{k+1} -\textbf{u}_k \leq \dot{\textbf{u}}^{max},  &\text{  }\forall k \in \mathcal{T}  \\
 &\textbf{s}^{min}_{k+1} \leq \textbf{s}_{k+1} \leq \textbf{s}^{max}_{k+1},  &\text{  }\forall k \in \mathcal{T}
\end{array}
\end{split}
\label{equ:mpc}\end{aligned}$$ where $Q \in \mathbb{R}^{5\times 5}$, $R \in \mathbb{R}^{2\times 2}$ are both positive definite weighting matrices for tuning, $\textbf{s}^{ref}_{k+1}$ is the reference trajectory, $\textbf{s}^{min}_{k+1}$ and $\textbf{s}^{max}_{k+1}$ are the bounds of each state: $$\begin{aligned}
    \textbf{s}^{min}_{k+1} &= [x^{min}_{k+1}, -w_{lane}, -\pi / 2, 0, 1]^\mathsf{T}, \\
    \textbf{s}^{max}_{k+1} &= [x^{max}_{k+1}, w_{lane}, \pi / 2, v^{max}, 3]^\mathsf{T},\end{aligned}$$ where $w_{lane}$ represents the width of each lane.

This program has standard solvers [@David2015Linear]. The state bounds in are used to guarantee there is no overlapping between all vehicles’ positions. Once received action from the DRL agent, this MPC program would evaluate whether it is feasible (safe) to implement this action. If it is not safe to change lane, it will continue to use a lane-keeping trajectory as a reference. Our work in [@han2019exploiting] shows how constraints in the state bounds are relaxed by sharing a sequence of future positions (this information is not required in behavior planning).

Distributed Learning
--------------------

In the centralized version of feedback deep Q-learning, all vehicles share one replay memory D and one action-value function $Q$. It means the training process needs support from a centralized cloud center. Also, the reward also needs this cloud center to calculate traffic flow and driving comfort. It can be simulated offline, but it is hard to implement this algorithm in reality because of the dependency of the cloud center. When the number of vehicles increases, the communication cost will increase dramatically. Therefore, a distributed learning algorithm is considered, where each vehicle has its own action-value function and learn it independently. The reward function for distributed DRL is limited to the information from its $\epsilon$-neighbors, defined as:

The local reward function for distributed DRL is $$R(s_v, s_f, a) = w \cdot s_v(t, l_k) +  \mathcal{C}_{single}(a_{it}, a),
\label{equ:reward_distributed}$$ where $a_{it}$ is the vehicle $i$’s acceleration at time $t$, $a$ is its action at state $s_v$ and $s_f$.

The reward function is a weighted sum of future velocity state and its own driving comfort. Passengers’ preference can determine this weight. A smaller $w$ is set for passengers who dislike frequent lane-changing; a larger $w$ is set for passengers who want to arrive at their destination faster. Because each vehicle conducts learning by itself, it can have different weight preferences.

The distributed version of the feedback deep Q-learning is shown in Alg. \[alg:distributed-qlearning\]. Though it does not need the support from a cloud center, its reward does not directly contribute to system-level traffic flow and driving comfort, and its convergence is slower than the centralized version. There is one experiment in section \[sec:experiment\] comparing the centralized and distributed versions.

Experiment {#sec:experiment}
==========

This section introduces the experiment results of three experiments:

-   Compare centralized learning with traditional control,

-   Compare distributed learning with centralized learning,

-   A case study for a freeway with road closures.

In these experiments, vehicles are randomly scattered on different lanes of a 1000-long freeway as their initial positions. All vehicles loop on this freeway. The total number of vehicles ranges from 100 to 900. For different traffic densities, experiments run for 4000 time steps. Traffic flow and driving comfort are evaluated based on the statistics in the last 1000 time steps. Also, the experiment runs 30 times under different initialization for each traffic density.

First Comparison
----------------

This experiment compares the centralized version of feedback deep reinforcement learning in Alg. \[alg:centralized-qlearning\] and a traditional control method. The traditional control used here is a control synthesis algorithm put forward in [@han2019exploiting]. Each autonomous vehicle is modeled as a hybrid system. The basic idea for behavior planning is to judge whether the benefit of lane changing is significant enough or not based on a pre-tuned threshold.

![The comparison between centralized learning and traditional control. Reinforcement learning is able to learn a better/comparable policy with larger traffic flow and better driving comfort.[]{data-label="fig:BP_result1"}](BP_result1.eps){width="3.4in"}

As shown in Fig. \[fig:BP\_result1\], the RL agent gets both larger traffic flow and better driving comfort when traffic density $\rho$ is low. When $\rho$ grows, the result of the RL agent gets worse, but it is still comparable with traditional control. The two dashed lines in this figure show two bounds for behavior planning. They consider two extreme cases: always keep lane and always change lane. When $\rho$ is low, lane-changing can increase traffic flow significantly while sacrificing driving comfort. However, lane-changing will only downgrade passengers’ experience rather than achieve a higher speed when the road is saturated. Consequently, the best choice is to keep a lane when $\rho$ is high.

Note that $\rho$ is not included in the state space, which means autonomous vehicles do not know the current traffic density. Intuitively, the optimal policy should be different under different traffic densities. In fact, excluding traffic density is an intentional design for the state space of distributed learning because vehicles cannot verify the current traffic density by limited information from its $\epsilon$-neighbors. Traffic density $\rho$ is excluded in Alg. \[alg:centralized-qlearning\] in order to keep the state space consistent between two algorithms. This experiment also shows the ability of the RL agent to adapt to different traffic densities by itself. When $\rho$ cannot be accurately identified by a single-vehicle, it is more reliable and stable for a traditional controller to use a fixed threshold for different $\rho$. However, it cannot adopt a better result for different traffic densities, as shown in Fig. \[fig:BP\_result1\].

Second Comparison
-----------------

![The comparison between different distributed learning and centralized learning. They only have a slight difference in most cases.[]{data-label="fig:BP_result2"}](BP_result2.eps){width="3.4in"}

This experiment compares distributed and centralized versions of feedback deep reinforcement learning in Alg. \[alg:centralized-qlearning\] and Alg. \[alg:distributed-qlearning\]. Intuitively, the centralized learning will have a better result, because its reward function is directly related to the evaluation criteria: traffic flow and driving comfort. Nevertheless, distributed learning is easier to implement because it does not rely on any cloud center. Therefore, we wonder to what extent the result will deteriorate. The result is shown in Fig. \[fig:BP\_result2\]. In most cases, the distributed learning has a slightly smaller traffic flow and slightly worse driving comfort. Though the difference becomes relatively larger sometimes, it is still acceptable.

Case Study
----------

![The scenario for the case study where there are two road closures on lane \#1.[]{data-label="fig:closed_lane"}](closed_lane.pdf){width="3in"}

This experiment shows a case study where there are two road closures on one lane. As shown in Fig. \[fig:closed\_lane\], the road closures may be caused by an accident, or they are under construction. These road closures both have a length of 200. We want to test whether the learning algorithm can work in a more complicated scenario. Autonomous vehicles are assumed to know the position of the road closures when it reaches the $\epsilon$-neighbor range of these segments. They can share this information with its $\epsilon$-neighbors (add as additional information to be shared in this case) or gather this information through V2I communication from infrastructure.

![The comparison between distributed learning and traditional control. The RL agent gets a comparable traffic flow while improving driving comfort a lot.[]{data-label="fig:BP_result3"}](BP_result3.eps){width="3.4in"}

The state value of for future velocity is defined to be 0 on the road closures. The result is shown in Fig. \[fig:BP\_result3\]. When $\rho$ is low, lane-changing can get a larger traffic flow while sacrificing driving comfort. Lane-keeping becomes a better choice when $\rho$ is high. Because the road closures are relatively long in this case, the critical point is small, and lane-keeping is a better strategy for most traffic densities. Though the RL agent makes more lane-changing, it avoids massive acceleration to provide better driving comfort.

conclusion {#sec:conclusion}
==========

This paper focuses on the behavior planning challenge for CAVs, and designs a policy learning algorithm that considers feedback from controllers and safety requirements with shared information to enhance operational performance. We also show the benefit of V2V communication. The problem is modeled as an MDP problem. The solution, feedback deep reinforcement learning, utilizes $T$-time-step future information shared among autonomous vehicles’ neighbors. The action of the proposed DRL algorithm receives feedback from a controller, which provides a safety guarantee. The centralized learning method can improve the overall system performance in terms of traffic flow and driving comfort, while the training process needs support from a cloud center. The distributed learning method is easier to implement online learning, but each vehicle only gets a local reward, and they do not share their historical transition experience. From experiment results, feedback DRL based behavior planning shows its advantages compared with a control synthesis algorithm. Distributed learning does not downgrade performance too much compared with centralized learning, which means it can be used in online learning. The case study with road closures shows the feedback DRL policy can be applied to a more complicated scenario. More experiments can be done to verify the scalability and robustness of the proposed state space design in the future.

Kinematic Bicycle Model {#sec:kinematic_model .unnumbered}
-----------------------

The control vector for this vehicle is defined as $ \textbf{u}_\tau \triangleq [ \delta_\tau, a_\tau ]$, where $a_\tau$ is its acceleration. The state vector is defined as $\textbf{s}_\tau \triangleq[x_\tau,y_\tau,\psi_\tau,v_\tau,l_\tau]$, where $l_\tau$ is the current lane number. The detail equations can be found in [@cesari2017scenario] as follows: $$\begin{aligned}
    \begin{cases}
    \dot{x}_\tau &= v_\tau cos(\psi_\tau + \beta_\tau), \\
    x_{\tau+1} &= x_\tau + \dot{x}_\tau \tau_{cs}, \\
    \dot{y}_\tau &= v_\tau sin(\psi_\tau + \beta_\tau), \\
    \tilde{y}_{\tau+1} &= y_\tau + \dot{y}_\tau \tau_{cs}, \\
    y_{\tau+1} &= \tilde{y}_{\tau+1} + \begin{cases} 
            -w_{lane}, & \text{if } \tilde{y}_{\tau+1} \geq \frac{w_{lane}}{2}; \\
             w_{lane}, & \text{if } \tilde{y}_{\tau+1} < -\frac{w_{lane}}{2};   \\
             0, & \text{otherwise}. \end{cases} \\
    \psi_{\tau+1} &= \psi_\tau + \frac{v_\tau cos \beta_\tau tan \delta_\tau}{l_f + l_r} \cdot \tau_{cs}, \\
    v_{\tau+1} &= v_\tau+ a_\tau \tau_{cs}, \\
    l_{\tau+1} &= l_\tau + \begin{cases} 
            1, & \text{if } \tilde{y}_{\tau+1} \geq \frac{w_{lane}}{2}; \\
            -1, & \text{if } \tilde{y}_{\tau+1} < -\frac{w_{lane}}{2};  \\
             0, & \text{otherwise}, \end{cases} 
    \end{cases} 
    \label{equ:kinematic}\end{aligned}$$ where $\beta_\tau$ is the slip angle. Once the control variable $\delta_\tau$ is determined, this angle can be updated as $\beta_\tau = tan^{-1} \left( \frac{l_r}{l_f+l_r} \cdot tan \delta_\tau \right)$. More compactly, the update of the state vector is denoted as $\textbf{s}_{\tau+1} = f( \textbf{s}_\tau , \textbf{u}_\tau)$ in the MPC program .

[^1]: This work was supported by NSF 1849246, NSF 1932250 grants. S. Han and F. Miao are with the Department of Computer Science and Engineering, University of Connecticut, Storrs Mansfield, CT, USA 06268. Email: {songyang.han, fei.miao}@uconn.edu.
