## RESC: A Reinforcement Learning Based Search-to-Control Framework for Quadrotor Local Planning in Dense Environments

Zhaohong Liu, Wenxuan Gao, Yinshuai Sun, and Peng Dong

Abstract -Agile flight in complex environments poses significant challenges to current motion planning methods, as they often fail to fully leverage the quadrotor dynamic potential, leading to performance failures and reduced efficiency during aggressive maneuvers. Existing approaches frequently decouple trajectory optimization from control generation and neglect the dynamics, further limiting their ability to generate aggressive and feasible motions. To address these challenges, we introduce an enhanced Search-to-Control planning framework that integrates visibility path searching with reinforcement learning (RL) control generation, directly accounting for dynamics and bridging the gap between planning and control. Our method first extracts control points from collision-free paths using a proposed heuristic search, which are then refined by an RL policy to generate low-level control commands for the quadrotor controller, utilizing reduced-dimensional obstacle observations for efficient inference with lightweight neural networks. We validate the framework through simulations and real-world experiments, demonstrating improved time efficiency and dynamic maneuverability compared to existing methods, while confirming its robustness and applicability.

## I. INTRODUCTION

Quadrotors are extensively used in a variety of applications, including rescue operations, fire and electricity inspection, and package delivery [1]. Achieving autonomous flight in complex environments necessitates effective motion planning, which is crucial for ensuring safe and efficient navigation. Current state-of-the-art motion planning algorithms for quadrotors primarily focus on generating collision-free and dynamic feasible trajectories, subsequently optimizing them to achieve smoothness and aggressiveness [2]. Time allocation plays a pivotal role in the trajectory generation process, as suboptimal time allocation can lead to inefficient trajectories and underutilization of the quadrotor dynamic capabilities [3].

However, trajectory optimization methods often fail to fully leverage the quadrotor dynamic potential and struggle to operate effectively in high-agility scenarios [4]. Furthermore, the trajectories generated by these optimizers must be tracked by a controller, introducing delays, increased computational overhead, and mismatches between the optimized and actual flight trajectories, particularly under aggressive maneuvers.

To address these challenges, we propose an improved Search-to-Control RL-based motion planner for quadrotor

This work was supported by the National Natural Science Foundation of China under Grants 61803260. ( Corresponding author: Peng Dong. )

The authors are with the School of Aeronautics and Astronautics, Shanghai Jiao Tong University, China (e-mail: zhliu25@sjtu.edu.cn; )

The source code will be released at: https://github.com/JaimeParker/rescpilot

Fig. 1: Our quadrotor flying in challenging indoor environments while avoiding obstacles.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-b4d03fc8-8340-43a6-999c-81385fbf5b4a.jpg)

autonomous flight in dense environments. Our approach aims to maximize the utilization of the quadrotor dynamic capabilities while relying solely on a rate controller. Our method begins with a path searching algorithm based on heuristic search, which constructs a collision-free, minimumlength path in a discrete grid space (Fig. 2). This high-level path is then converted into control points, which serve as input for the RL control generator. The RL policy network processes observations of the quadrotor state, control points, and environmental information to generate low-level control commands. Finally, the generated commands are sent directly to the quadrotor rate loop controller, enabling the execution of agile and precise motions.

In this letter, we design a R einforcement L E arning based S earch-toC ontrol framework called RESC . Compared to existing motion planners, our method departs from trajectory generation and optimization frameworks by directly generating control commands using a RL policy. This approach can handle discrete paths that may not inherently satisfy kinodynamic and non-holonomic constraints, producing aggressive, collision-free motions within specified dynamic limits, without relying on iterative numerical optimization. We evaluate the effectiveness and robustness of the proposed method through extensive simulations and real-world experiments across diverse scenarios. Our key contributions are summarized as follows:

- (1) A unified Search-to-Control planning framework that combines visibility-based path searching with RLdriven control generation, enabling aggressive and feasible motions without explicit trajectory representation or optimization.
- (2) A comprehensive RL environment that fully utilizes quadrotor dynamic potential to train policies, facilitat-

Fig. 2: Illustration of the proposed method. RL policy is used to generate control commands based on the quadrotor state, obstacles, and control points.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-6446f61c-adee-4e6e-9eb6-5bf9110c2f86.jpg)

ing the generation of low-level control commands for diverse environments and seamless integration with the pathfinding module.

- (3) A reduced-dimensional obstacle observation method that efficiently captures environmental complexity, enabling lightweight neural network training and inference for robust control generation.
- (4) Extensive simulation and real-world experiments on an autonomous quadrotor, demonstrating the robustness and practical applicability of the proposed framework. The source code is released for further research.

## II. RELATED WORK

## A. Trajectory Optimization Methods

Trajectory optimization methods prevail as the predominant approach for quadrotor motion planning, which focus on generating smooth, dynamically feasible, and collisionfree trajectories by modeling as optimal control problems. The pioneering approach is the minimum-snap trajectory generation [5] in differential flatness outputs, representing the trajectory as a polynomial and using gradient descent to iteratively modify the time segments. Several works propose two-stage solving methods where trajectory generation occurs in the second stage [6]-[11]. For instance, a safe flight corridor composed of convex polyhedra is introduced in [6], providing constraints for optimization. Additionally, B-splines are utilized for kinodynamic path searching in [8], where an elastic optimization method is employed to refine trajectories. The Euclidean Signed Distance Field (ESDF) map is employed in [12] to obtain collision potentials and in [9] to find a path in the velocity field, improving time allocation. [10] proposes a comprehensive pipeline for quadrotor motion planning, including kinodynamic path searching, Bspline trajectory generation, time allocation, and nonuniform B-spline optimization based on ESDF. Subsequent works [13] and [14] introduce topological path and risk-aware yaw planning. Given the computational intensity of constructing ESDF, [15] proposes an ESDF-free gradient-based planner. Wang et al. [16] focuses on spatial-temporal deformation, which aligns with our approach by considering both geometric and temporal planning concurrently. These methods effectively generate smooth, safe, and feasible trajectories.

However, they may be conservative regarding the ignorance of quadrotor dynamic capabilities.

## B. RL based Methods

RL-based methods have gained significant traction in quadrotor motion planning due to their capability to optimize non-convex and discontinuous objectives. In [17], the authors propose an imitation learning (IL)-based teacherstudent pipeline that directly maps noisy sensory depth images to polynomial trajectories. While IL is quite efficient, its scalability is limited by the need for expert data collection. The work in [18] utilizes a topological pathsearching algorithm to generate guiding paths, which are subsequently refined by reinforcement learning to produce control commands. This two-stage method performs well in selected scenarios, outperforming some traditional methods. Song et al. [19] extracts features from depth images and uses a teacher-student IL framework similar to [17]. Furthermore, it generates control commands directly and includes a perception reward to guide yaw planning. In [4], [20], Song et al. investigates the strengths and limitations of optimal control and RL, proposing that RL is advantageous as it can optimize better objectives that may be infeasible for optimal control methods to address. In [21], Zhao et al. propose a hierarchical RL framework where an outer-loop policy dynamically configures speed and acceleration constraints based on online observations, significantly improving flight efficiency and safety in cluttered environments.

However, these methods frequently rely on imitation learning (IL) with expert knowledge or are trained in fixed environments, limiting their generalization and transferability to unknown environments, thereby hindering autonomous flight in new scenarios. Instead of using RL for the entire planning process, we employ it as an optimizer to refine control points and directly generate control commands. This strategy allows for more efficient learning and better adaptation to complex environments.

## III. SYSTEM OVERVIEW

## A. Planning Structure

The proposed framework, illustrated in Fig. 3, follows a planning process that consists of two stages. First, the Visibility Path Searching algorithm constructs a reference path from the start to the goal, ensuring obstacle avoidance and path length minimization. This reference path is discretized into control points based on the current position and constraints among control points (Section IV). Second, the RL Control Generation method is employed to generate control commands that optimize the quadrotor motion. These commands guide the quadrotor to follow the control points within a specified tolerance, determining the proximity requirements and strategies for obstacle avoidance in continuous space, as detailed in Section V.

## B. Quadrotor dynamics

The quadrotor is modelled as a rigid body governed by a linear system with decoupled translational and rotational

Fig. 3: An overview of our proposed framework. It focuses exclusively on control generation during training. During deployment, it integrates state estimation and mapping data to generate control commands through a two-step process.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-faf0b785-2aa8-4669-99e5-369d2fa992da.jpg)

dynamics. The control input vector u ∈ R 4 corresponds to the individual rotor thrusts, specifically defined as u = [ f , f 1 2 , f 3 , f 4 ] T with f i denotes the thrust scalar of the i -th rotor. The complete state of the quadrotor is defined as x = [ p w , ˙ p w , ¨ p w , Θ , ω b ] , where p w ∈ R 3 is the position in world frame, Θ = [ ϕ, θ, ψ ] T ∈ SO 3 represents roll, pitch and yaw angle, and ω b denotes angular velocity in body frame.

The dynamics equations are

$$\ddot { p } _ { w } = - g e _ { 3 } + \sum _ { i = 0 } ^ { 3 } \frac { f _ { i } } { m } R _ { b } ^ { w } b _ { 3 }. \quad \quad ( 1 ) \quad \text{plan} \\ \text{corn} \quad \text{corn} \quad \text{.}$$

$$J \dot { \omega } ^ { b } = - \omega ^ { b } \times J \omega ^ { b } + G _ { a } + \tau. \quad \ \ ( 2 ) \stackrel { \omega y \ a } { \ r e d u }$$

where v w is velocity, R w b is rotation matrix from body frame to world frame, b 3 is z axis in body frame, f i is thrust of each rotor, m is mass, J is diagonal inertia matrix, , G a and τ = [ τ x , τ y , τ z ] T is gyroscopic torque and torque by thrusts respectively. We use Euler angles to represent rotation and set a limit on roll and pitch angles to avoid singularity.

The control allocation matrix that can be used to get relation between thrusts and torques is given by

$$\begin{bmatrix} 1 & 1 & 1 & 1 \\ - l / \sqrt { 2 } & - l / \sqrt { 2 } & l / \sqrt { 2 } & l / \sqrt { 2 } \\ - l / \sqrt { 2 } & l / \sqrt { 2 } & l / \sqrt { 2 } & - l / \sqrt { 2 } \\ - \kappa & \kappa & - \kappa & \kappa \end{bmatrix} u = \begin{bmatrix} f _ { T } \\ \tau _ { x } \\ \tau _ { y } \\ \tau _ { z } \end{bmatrix}. \quad ( 3 ) & \text{other} \\ & \text{cursive} \\ & \text{reduce}$$

where κ denotes rotor's torque coefficient, and l represents the distance from the center of mass to the rotor center.

## IV. VISIBILITY PATH SEARCHING

Our path searching algorithm combines elements of the A*, jump point search (JPS) [22], and visibility graph methods. We inherit the heuristic function and the open and

## Algorithm 1: Visibility Path Searching.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-242a798b-06ca-4b3d-9e3f-db77430eb5d2.jpg)

closed list architecture from the A* algorithm but implement an alternative cost function. Elements of the JPS method are incorporated to reduce the number of nodes in the open list, minimizing computational costs associated with sorting these nodes. Additionally, we leverage the optimal distance characteristics of the visibility graph method without constructing complex polygon edges and vertices.

The proposed strategy facilitates omnidirectional movement through obstacle corner detection and visibility checks, rather than restricting movement to increments of π/ 4 in 2D plane. This approach is implemented by selecting obstacle corners in the Corner () function and assessing voxel visibility in the VisibleCheck () function. As a result, this method reduces the computational and memory overhead typically required for accessing each vertex and edge of the voxels. Subsequently, the corners are linked to form paths, which are then sampled to generate control points. The detailed algorithm is presented in Algorithm 1.

## A. Obstacle Corner Search

We define an obstacle corner as a voxel with only one occupied neighbor, indicating its location at the edge of an obstacle. In the context of the Visibility Graph, the optimal path from the start to the goal point traverses vertices that represent the corners of obstacles. Therefore, obstacle corners Corner () , including the start and goal points, are exclusively utilized as elements in the open list. This approach reduces the number of nodes in the open list and decreases sorting costs, similar to JPS, while maintaining the optimal path characteristics of the Visibility Graph.

The cost of a obstacle corner using EuclideanCost () is defined as

$$g ( n ) = g ( n - 1 ) + | | p _ { n } - p _ { n - 1 } | |. \quad \quad ( 4 )$$

## B. Visibility Expansion

When a new obstacle corner is detected, the Bresenham algorithm [23] is employed to perform a visibility check using VisibleCheck () , determining whether the corner is visible from the current node. Only corners confirmed to be visible are considered as child nodes and candidates for the open list, ensuring that a connected path of vertices via line segments is traced from start to goal. We adopt an expansion method from Hybrid A* [24] in RandomVisibleCheck () to verify the existence of a collision-free path between the current node and the goal along a line segment. The invocation frequency of this function increases as the distance to the goal decreases.

## V. RL BASED CONTROL GENERATION

## A. Policy Framework

We frame the control point optimization and control generation problem within a reinforcement learning (RL) framework. The quadrotor agent selects actions based on the policy π ( a s t | t ) , transitioning from state s t to s t +1 according to the dynamics-kinematics update of the system. The agent receives a reward R ( s , a , s t t t +1 ) that guides learning. Detailed configurations of the state space, action space, and reward function will be discussed in subsequent sections.

1) Observation and action space: The observation space contains three main components: the state of the quadrotor, control points and obstacle information. The state vector is defined as

$$s = \left [ p _ { w } ( t ), v _ { w } ( t ), \omega _ { b } ( t ), \Theta ( t ), p _ { \text{cp} }, O _ { \text{obs} } \right ], \quad ( 5 )$$

where p cp represents control points, and O obs denotes obstacle information, .

The observation vector is derived from the state vector, directly incorporating components such as v w ( ) t , ω b ( ) t , Θ ( ) t , and O obs . Other elements of the state are converted into lower-level features, making them more generalized and easier for policy training. For example, we use relative position from quadrotor position to control points instead of absolute position. Additionally, a normalized vector representing the x-axis direction of body frame in world frame ( x w b ) is used to guide the yaw to stay visiualized for control point based on the field-of-view (FOV) of depth camera. Obstacle observation O obs will be discussed in the following section.

The action produced by RL policy is a ( ) = [ t f T , ω req b ] , where f T is the collective thrust and ω req b represents the desired body rate. This control approach has been proved effective for learning-based quadrotor control and planning tasks [25]. Although commonly used in drone racing tasks due to its low latency and high frequency, it can present a gap between simulation and real-world performance. Despite this limitation, this control approach leads to smoother acceleration and velocity, improving energy efficiency and safety compared to higher-dimensional control approaches.

Fig. 4: Visualization of our RL training environment. The environment is reset when collision or the last control point is reached.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-8fb16419-6792-47b2-b62f-87eed3f16c34.jpg)

2) Reward function: According to paper [20], RL outperforms Optimal Control (OC) in terms of optimizing a better objective that OC may not be able to solve. From this perspective, we design a reward function that explicitly represents agile motion, collision avoidance, and smoothness in control. The reward function is defined as

$$\text{es} \quad r ( t ) = k _ { p } r _ { p } + r _ { c } + k _ { d } r _ { d } + k _ { v } r _ { v } + k _ { s } | | \omega _ { b } | | + r _ { f }, \quad ( 6 )$$

where r p denotes the customized progress reward, represents the penalty for collisions, r d is the penalty for dynamic violations, r v is the penalty for FOV violations, and r f rewards task completion.

Specifically, the progress reward r p ( ) t is defined as

$$r _ { p } ( t ) = \frac { | | p _ { \text{int} } - p _ { w, i - 1 } | | - | | p _ { \text{int} } - p _ { w, i } | | } { v _ { \max } \Delta t } + r _ { t }, \quad ( 7 )$$

where p int denotes the intersection point if the vector p w,i -p w,i -1 intersects the control point area; otherwise, the intersection point is directly p cp ,i . We normalize the progress reward and subsequently introduce a negative term r t , thereby imposing a time penalty which maximizing the utilization of the quadrotor dynamic capabilities.

For dynamic violation penalty r d , we impose limits on acceleration and velocity to ensure dynamic smoothness and safety. Additionally, attitude limits are set to prevent overly aggressive maneuvers and avoid singularities.

$$r _ { d } ( t ) = - n, \ \ n \in \{ 0, 1, 2, 3 \}, \quad \quad ( 8 )$$

where n represents the number of elements from the set [ v w ( ) t , a w ( ) t , Θ ( )] t that exceed their prescribed limits.

The FOV violation penalty r v is set to -1 if neither of the next two control points is within the FOV of the depth camera. This ensures that the quadrotor maintains visibility of the control points.

## B. Training methodology

We create a training environment based on OpenAI gym [26], providing an interface for creating a customized environment. At the beginning of each episode, the velocity, body rate, orientation, control points and obstacles are randomly

Fig. 5: Example of Kinematic Pseudo-Raycast. The blue region represents the guided raycast from the quadrotor position to the control point, while the red region indicates the guided raycast from the quadrotor position to the velocity control point.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-1951b3e3-0040-4aaf-9355-b5e3be8dcf6b.jpg)

initialized following a predefined probabilistic pattern, while the position if fixed at the center of the map (Fig. 4).

1) Environment randomization: The environment is initialized with control points and obstacles, with the quadrotor starting at the map center. The first control point is randomized within a specified distance from the quadrotor position. Subsequent control points are placed within a defined distance and angle from the previous one. Velocity and orientation constraints are applied to keep the quadrotor motion stable, ensuring its velocity remains basically towards the first control point, and its pitch, roll, and yaw stay within specified limits.

Obstacles are placed along or near the lines connecting the quadrotor to the control points, ensuring collision-free paths while allowing some proximity to obstacles, rather than creating a maze-like environment.

2) Control point observation: Following Song et al. [4], we observe two relative positions of the control points. For the last control point, the relative position of the quadrotor to this point is repeated. We define a finite-height column with a radius of d hor and a height of 2 d hgt to represent the area within which the control point is considered passed. The observation is then updated to the next two control points. With this design and the randomization of the environment, the use of a small number of control points (we use two in our training) enables efficient training while allowing generalization to more complex tasks with a larger number of control points.

3) Obstacle observation: We introduce a novel method to describe obstacle information efficiently, called Kinematic Guided Pseudo-Raycast and ESDF Perception (KGP-ESDF), as shown in Fig. 5.

The core concept of KGP is to identify the most relevant obstacle edges for the quadrotor motion towards the control point, simultaneously avoiding collisions along the current velocity direction. First, the algorithm performs a sectorbased search along the line from the current position p w to the control point to identify the nearest obstacle edges, denoted as o cp . Subsequently, a similar search is conducted along the line from p w to the velocity control point p vcp as defined by the equation

$$p _ { v c p } = p _ { w } + \frac { v _ { w } } { | | v _ { w } | | } \cdot | | p _ { w } - p _ { c p, i } | |, \quad \quad ( 9 )$$

to identify the nearest obstacle edges, denoted as o vcp .

The relative positions of these edge points are incorporated as part of the observation. Additionally, we record the distance d vel from the current position to the velocity control point, which represents the collision-free distance.

We generate o sdf to represent the surrounding obstacle information, derived from the ESDF map. Finally, the obstacle observation O obs is constructed as

$$O _ { o b s } = \left [ o _ { c p }, o _ { \text{vcp} }, d _ { \text{vel} }, o _ { \text{sdf} } \right ]. \quad \quad ( 1 0 )$$

## C. Sim-to-real transfer

Transferring a policy trained in a customized environment to simulation and then to a real-world platform is challenging due to factors like inaccurate modeling of quadrotor dynamics, system delays, low-level controller errors, and action oscillations introducing high-order system complexities. We implement strategies to mitigate these issues.

We first train the policy in a customized Gym-based environment before transferring it to Gazebo simulation with PX4 firmware and then testing it on our quadrotor platform without fine-tuning. We meticulously identified the Iris drone and our quadrotor models, achieving a relatively accurate representation. To ensure compatibility with lowlevel control, we utilize the PX4 multicopter rate controller and control allocator to compute thrusts from the desired body rate and collective thrust outputs of the RL policy.

Additionally, we apply domain randomization and action delay queue during RL training to enhance policy robustness and prevent overfitting, compensating for numerical integration errors and modeling discrepancies between simulation and reality. We randomized the quadrotor physical properties, including mass, inertia, and thrust mapping, and ignored battery voltage drop and motor heating effects due to short flight duration.

## VI. EXPERIMENT RESULTS

## A. Training details

The policy is trained using the Proximal Policy Optimization (PPO) [27] algorithm in Stable-Baselines3 [28], which has demonstrated strong performance in various continuous control tasks, including drone racing. We train two separate models for scenarios with and without obstacles to streamline training and improve efficiency. This approach also mitigates the issue of forgetting, which we encountered when sequentially training a single model first on obstacle-free and then on obstacle-present cases. The decision to use a specific model depends on the values from the ESDF map and the camera's ideal range.

The quadrotor physical properties and the RL algorithm's hyperparameters are listed in Table II. Following previous research, we employ Runge-Kutta 4th order integration to update the quadrotor state, excluding aerodynamic drag during training due to its insignificance for the current v max.

Fig. 6: Visualization of the planning comparison in 4 scenes between baseline and proposed methods.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-d971e56e-c346-4e64-ba3a-77add90c1170.jpg)

TABLE I: Planners comparison in four scenes with respect to time span and energy consumption.

|                   | Time Span ( s )   | Time Span ( s )   | Time Span ( s )   | Time Span ( s )   | Energy ( m 2 /s 5 )   | Energy ( m 2 /s 5 )   | Energy ( m 2 /s 5 )   | Energy ( m 2 /s 5 )   |
|-------------------|-------------------|-------------------|-------------------|-------------------|-----------------------|-----------------------|-----------------------|-----------------------|
| Method            | scene1            | scene2            | scene3            | scene4            | scene1                | scene2                | scene3                | scene4                |
| Fast-Planner [10] | 7.43              | 6.03              | inf               | 5.33              | 176.55                | 44.36                 | 0                     | 151.81                |
| EGO-Planner [15]  | 4.64              | 4.88              | inf               | 4.24              | 824.03                | 497.84                | 0                     | 578.94                |
| RESC              | 4.5               | 4.48              | 11.7              | 4.00              | 697.11                | 682.53                | 1147.1                | 611.52                |

TABLE II: Parameter of our method

|        | Variable                           |    Value | Variable                        | Value           |
|--------|------------------------------------|----------|---------------------------------|-----------------|
| IMP250 | m [kg] i xx [g m - ] i yy [g m - ] |    1.64  | κ [-] l x [m] l ys [m] l yl [m] | 0.012 0.88 0.88 |
| IMP250 |                                    |    0.011 |                                 |                 |
| IMP250 |                                    |    0.01  |                                 |                 |
| IMP250 | i zz [g m - ]                      |    0.007 |                                 | 0.88            |
|        | a max [m s - 2 ]                   |    6     | v max [m s - 1 ]                | 3.0             |
|        | k p [-]                            |    2     | k d [-]                         | 5.0             |
|        | k v [-]                            |    5     | k s [-]                         | 0.01            |
|        | r c [-]                            | -600     | r f [-]                         | 300             |
|        | d hor [m]                          |    0.5   | d hgt [m]                       | 0.2             |
|        | ∆ t [s]                            |    0.02  | r t [-]                         | -1.5            |
|        | n steps [-]                        | 2048     | γ [-]                           | 0.99            |
|        | batch size [-]                     |  128     | learning rate [-]               | 0.0003          |
|        | n epochs                           |   10     | λ (GAE)                         | 0.95            |

All policies are trained on a desktop computer equipped with an NVIDIA RTX 4090 GPU and an Intel i9-13900K CPU.

## B. Analysis and Comparison

We compare our method with Fast-Planner [10] and EGOPlanner [15] in terms of time to reach the same position in four different scenes(Fig. 6). To ensure fairness, we remove control disturbances and initialize both methods from the same state. The baseline methods use the same maximum velocity ( 3 m/s ), acceleration ( 6 m/s 2 ), and inflated obstacle size ( 0 3 . m ) as our method. Additionally, we impose stricter limits on z-axis position ( 1 ± 0 3 . m ), sensing horizon ( 5 0 . m ), and searching horizon ( 7 5 . m ) to align with our method. Other parameters are set to default values. The comparison of baseline methods and our method is shown in Table I.

In scenes 1, 2, and 4, our method outperforms the baseline approaches in terms of time efficiency, demonstrating its ability to generate more aggressive control commands. The drone achieves the desired motion accurately after executing the control commands, benefiting from the integration of dynamics in our policy network. In scene 3, only our method successfully reaches the goal, while the baseline methods fail to achieve the same result under the given acceleration and velocity constraints.

Consequently, our method shows better performance in terms of time efficiency, although it results in higher energy consumption compared to the baseline methods, primarily due to the aggressive control commands generated by the RL policy.

## C. Simulation and Real-world Experiments

We present several experiments conducted in both simulation and real-world environments to evaluation the feasibility and performance of the proposed method. In simulation, we focus on navigating a quadrotor through some cluttered and randomized environments, where the RL policy generates control inputs that drive the quadrotor from the start point to the goal with control points provided by proposed searching method while avoiding obstacles. In the simulation environment, as shown in Fig. 8, the quadrotor follows control commands generated by RL policy, demonstrating the feasibility and real-time performance of our method. The actual trajectory driven by the RL policy is visualized in red.

To validate the performance of proposed planning system in real-world conditions, we conducted experiments in an indoor environment. The quadrotor's pose data is provided by the NOKOV motion capture system, and the ESDF map

Fig. 7: Visualization of a indoor planning experiment over a short period of time with actual velocity and body rate profile.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-a888822f-7b17-4f37-9971-b25f9a25a3f1.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-04ff9d32-5b93-45fd-804a-0bf78a9549e5.jpg)

Fig. 8: Visualization of the actual trajectory driven by the RL policy in the simulation environment.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-78dbbd2b-acff-4267-8284-7dc1e71f1945.jpg)

is generated using the pose and depth images. As depicted in Fig. 9, the quadrotor successfully navigates around obstacles and adjusts its attitude to reach the goal. In Fig. 10, we show the actual trajectory of the quadrotor during a flight, further confirming the effectiveness of the RL policy in guiding the quadrotor to the goal while avoiding obstacles.

These experiments demonstrate the robustness and adaptability of the RL policy across both simulated and realworld environments, highlighting its potential for practical deployment in autonomous quadrotor motion planning tasks.

Fig. 9: Real-world experiment conducted in an indoor environment: (a) The quadrotor successfully passes the obstacle. (b) The quadrotor adjusts its attitude to navigate toward the goal. (c) Composite image showing the sequence of the flight. (d) Desired throttle, rate and actual rate of the quadrotor.

## VII. CONCLUSION

In this letter, we present an improved framework for quadrotor local planning that integrates path searching and control generation using visibility-based methods and reinforcement learning. Our approach utilizes visibility path searching to generate safe control points with minimal path length, which are subsequently optimized by an RL policy to produce desired rate and throttle commands. We validate our method in several simulated environments and compare its performance against baseline approaches, showing superior

Fig. 10: Trajectory of an indoor experiment and our quadrotor.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00275v4-figure-86983c97-1fee-47a7-bf5f-b6db59715859.jpg)

time efficiency. We further conduct real-world experiments to demonstrate the feasibility and effectiveness of our method in practical applications.

Future work will focus on optimizing the RL policy to reduce energy consumption, enhance risk awareness through active yaw planning, and improve the success rate. Additionally, we aim to extend our method to outdoor real-world scenarios and expand it to fully 3D space. We also plan to investigate the potential of our approach in highly dynamic environments.

## REFERENCES

- [1] H. Wang, H. Li, B. Zhou, F. Gao, and S. Shen, 'Impact-aware planning and control for aerial robots with suspended payloads,' IEEE Transactions on Robotics , vol. 40, pp. 2478-2497, 2024.
- [2] L. Quan, L. Han, B. Zhou, S. Shen, and F. Gao, 'Survey of uav motion planning,' IET Cyber-Systems and Robotics , vol. 2, no. 1, pp. 14-21, 2020. [Online]. Available: https://ietresearch.onlinelibrary. wiley.com/doi/abs/10.1049/iet-csr.2020.0004
- [3] F. Gao, W. Wu, J. Pan, B. Zhou, and S. Shen, 'Optimal time allocation for quadrotor trajectory generation,' in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2018, pp. 47154722.
- [4] Y. Song, M. Steinweg, E. Kaufmann, and D. Scaramuzza, 'Autonomous drone racing with deep reinforcement learning,' in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2021, pp. 1205-1212.
- [5] D. Mellinger and V. Kumar, 'Minimum snap trajectory generation and control for quadrotors,' in 2011 IEEE International Conference on Robotics and Automation , 2011, pp. 2520-2525.
- [6] S. Liu, M. Watterson, K. Mohta, K. Sun, S. Bhattacharya, C. J. Taylor, and V. Kumar, 'Planning dynamically feasible trajectories for quadrotors using safe flight corridors in 3-d complex environments,' IEEE Robotics and Automation Letters , vol. 2, no. 3, pp. 1688-1695, 2017.
- [7] F. Gao, Y. Lin, and S. Shen, 'Gradient-based online safe trajectory generation for quadrotor flight in complex environments,' in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2017, pp. 3681-3688.
- [8] W. Ding, W. Gao, K. Wang, and S. Shen, 'An efficient b-spline-based kinodynamic replanning framework for quadrotors,' IEEE Transactions on Robotics , vol. 35, no. 6, pp. 1287-1306, 2019.
- [9] F. Gao, W. Wu, Y. Lin, and S. Shen, 'Online safe trajectory generation for quadrotors using fast marching method and bernstein basis polynomial,' in 2018 IEEE International Conference on Robotics and Automation (ICRA) , 2018, pp. 344-351.
- [10] B. Zhou, F. Gao, L. Wang, C. Liu, and S. Shen, 'Robust and efficient quadrotor trajectory generation for fast autonomous flight,' IEEE Robotics and Automation Letters , vol. 4, no. 4, pp. 3529-3536, 2019.
- [11] J. Tordesillas, B. T. Lopez, M. Everett, and J. P. How, 'Faster: Fast and safe trajectory planner for navigation in unknown environments,' IEEE Transactions on Robotics , vol. 38, no. 2, pp. 922-938, 2022.
- [12] H. Oleynikova, M. Burri, Z. Taylor, J. Nieto, R. Siegwart, and E. Galceran, 'Continuous-time trajectory optimization for online uav replanning,' in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2016, pp. 5332-5339.
- [13] B. Zhou, F. Gao, J. Pan, and S. Shen, 'Robust real-time uav replanning using guided gradient-based optimization and topological paths,' in 2020 IEEE International Conference on Robotics and Automation (ICRA) , 2020, pp. 1208-1214.
- [14] B. Zhou, J. Pan, F. Gao, and S. Shen, 'Raptor: Robust and perceptionaware trajectory replanning for quadrotor fast flight,' IEEE Transactions on Robotics , vol. 37, no. 6, pp. 1992-2009, 2021.
- [15] X. Zhou, Z. Wang, H. Ye, C. Xu, and F. Gao, 'Ego-planner: An esdffree gradient-based local planner for quadrotors,' IEEE Robotics and Automation Letters , vol. 6, no. 2, pp. 478-485, 2021.
- [16] Z. Wang, X. Zhou, C. Xu, and F. Gao, 'Geometrically constrained trajectory optimization for multicopters,' IEEE Transactions on Robotics , vol. 38, no. 5, pp. 3259-3278, 2022.
- [17] A. Loquercio, E. Kaufmann, R. Ranftl, M. M¨ uller, V. Koltun, and D. Scaramuzza, 'Learning high-speed flight in the wild,' Science Robotics , vol. 6, no. 59, p. eabg5810, 2021. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.abg5810
- [18] R. Penicka, Y. Song, E. Kaufmann, and D. Scaramuzza, 'Learning minimum-time flight in cluttered environments,' IEEE Robotics and Automation Letters , vol. 7, no. 3, pp. 7209-7216, 2022.
- [19] Y. Song, K. Shi, R. Penicka, and D. Scaramuzza, 'Learning perception-aware agile flight in cluttered environments,' in 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023, pp. 1989-1995.
- [20] Y. Song, A. Romero, M. M¨ uller, V. Koltun, and D. Scaramuzza, 'Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,' Science Robotics , vol. 8, no. 82, p. eadg1462, 2023. [Online]. Available: https://www.science.org/doi/abs/10.1126/ scirobotics.adg1462
- [21] G. Zhao, T. Wu, Y. Chen, and F. Gao, 'Learning speed adaptation for flight in clutter,' IEEE Robotics and Automation Letters , vol. 9, no. 8, pp. 7222-7229, 2024.
- [22] D. Harabor and A. Grastien, 'Online graph pruning for pathfinding on grid maps,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 25, no. 1, pp. 1114-1119, Aug. 2011. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/7994
- [23] J. E. Bresenham, 'Algorithm for computer control of a digital plotter,' IBM Systems Journal , vol. 4, no. 1, pp. 25-30, 1965.
- [24] D. Dolgov, S. Thrun, M. Montemerlo, and J. Diebel, 'Path planning for autonomous vehicles in unknown semi-structured environments,' The International Journal of Robotics Research , vol. 29, no. 5, pp. 485-501, 2010. [Online]. Available: https: //doi.org/10.1177/0278364909359210
- [25] E. Kaufmann, L. Bauersfeld, and D. Scaramuzza, 'A benchmark comparison of learned control policies for agile quadrotor flight,' in 2022 International Conference on Robotics and Automation (ICRA) , 2022, pp. 10 504-10 510.
- [26] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, 'Openai gym,' 2016. [Online]. Available: https://arxiv.org/abs/1606.01540
- [27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, 'Proximal policy optimization algorithms,' 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
- [28] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, 'Stable-baselines3: Reliable reinforcement learning implementations,' Journal of Machine Learning Research , vol. 22, no. 268, pp. 1-8, 2021. [Online]. Available: http://jmlr.org/papers/ v22/20-1364.html
Zhaohong Liu Wenxuan Gao Yinshuai Sun Peng Dong
RESC: A Reinforcement Learning Based Search-to-Control Framework for Quadrotor Local Planning in Dense Environments
Agile flight in complex environments poses significant challenges to current
motion planning methods, as they often fail to fully leverage the quadrotor
dynamic potential, leading to performance failures and reduced efficiency
during aggressive maneuvers.Existing approaches frequently decouple trajectory
optimization from control generation and neglect the dynamics, further limiting
their ability to generate aggressive and feasible motions.To address these
challenges, we introduce an enhanced Search-to-Control planning framework that
integrates visibility path searching with reinforcement learning (RL) control
generation, directly accounting for dynamics and bridging the gap between
planning and control.Our method first extracts control points from
collision-free paths using a proposed heuristic search, which are then refined
by an RL policy to generate low-level control commands for the quadrotor
controller, utilizing reduced-dimensional obstacle observations for efficient
inference with lightweight neural networks.We validate the framework through
simulations and real-world experiments, demonstrating improved time efficiency
and dynamic maneuverability compared to existing methods, while confirming its
robustness and applicability.