## SF-TIM: A Simple Framework for Enhancing Quadrupedal Robot Jumping Agility by Combining Terrain Imagination and Measurement

Ze Wang 1 4 , , Yang Li , Long Xu , Hao Shi , Zunwang Ma , Zhen Chu , Chao Li , 4 3 1 4 4 4 3 2 1

Fei Gao , Kailun Yang , and Kaiwei Wang

Abstract -Dynamic jumping on high platforms and over gaps differentiates legged robots from wheeled counterparts. Compared to walking on rough terrains, dynamic locomotion on abrupt surfaces requires fusing proprioceptive and exteroceptive perception for explosive movements. In this paper, we propose SF-TIM (Simple Framework combining Terrain Imagination and Measurement), a single-policy method that enhances quadrupedal robot jumping agility, while preserving their fundamental blind walking capabilities. In addition, we introduce a terrain-guided reward design specifically to assist quadrupedal robots in high jumping, improving their performance in this task. To narrow the simulation-to-reality gap in quadrupedal robot learning, we introduce a stable and high-speed elevation map generation framework, enabling zeroshot simulation-to-reality transfer of locomotion ability. Our algorithm has been deployed and validated on both the small/large-size quadrupedal robots, demonstrating its effectiveness in real-world applications: the robot has successfully traversed various high platforms and gaps, showing the robustness of our proposed approach. A demo video has been made available at https://flysoaryun.github.io/SF-TIM .

## I. INTRODUCTION

With the rapid development of legged robotics, quadrupedal robots have become essential in exploration and search and rescue missions due to their superior terrain passability [1]-[6]. Unlike wheeled robots, quadrupedal robots excel in handling complex terrains but are relatively difficult to control [7]. Reinforcement Learning (RL) algorithms utilizing proprioceptive sensor data, such as Inertial Measurement Units (IMU) and joint encoders, can enhance the terrain adaptability of quadrupedal robots, enabling them to traverse both smooth and rugged terrains and navigate stairs [2]. However, relying solely on proprioceptive sensors does not enable quadrupedal robots to perform vertical or horizontal jumping maneuvers, which are crucial for enhanced terrain traversal. Typically, these robots rely on exteroceptive sensors such as depth cameras

This was supported in part by Zhejiang Provincial Natural Science Foundation of China (Grant No. LZ24F050003), the National Natural Science Foundation of China (Grant No. 12174341), the National Key RD Program (Grant 2022YFB4701400), the 'Leading Goose' R&amp;D Program of Zhejiang (Grant No.2023C01177) and in part by Hangzhou SurImage Technology Co. Ltd. (Corresponding author: Kaiwei Wang.)

1 State Key Laboratory of Modern Optical Instrumentation, Zhejiang University, China

2 School of Robotics and National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China

3 State Key Laboratory of Industrial Control Technology, Zhejiang University, China

4 DeepRobotics Co. Ltd., China

Email: wangkaiwei@zju.edu.cn.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-1688056e-a7df-45f8-86ae-53a4c556b137.jpg)

(a) Lite3 jumps over the imaginary gap.

(b) X3O jumps over the real gap.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-e86c320f-de54-4cb6-8eae-db74c67c405e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-0dfa12ae-5314-4776-81d2-ddb48d0c3577.jpg)

(c) Lite3 jumps onto the height.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-e23dfc97-4b9c-421f-8594-682f18c3d1c5.jpg)

Fig. 1. Jumping experiment of Lite3 and X30 robots. The Lite3 and X30 robots perform horizontal and vertical jumps respectively, with the Lite3 robot jumping over an imaginary gap and the X30 over a real gap.

or LiDAR [1] to achieve jumping capabilities and further improve their terrain traversal abilities.

Currently, many quadrupedal robot control frameworks rely on exteroceptive sensors like depth cameras and LiDAR. Depth-camera-based frameworks [8], [9] often employ a teacher-student network approach. Initially, the teacher network is trained using elevation maps to learn the environmental features. Subsequently, the student network is trained using depth maps under the guidance of the teacher network, transferring learned knowledge through a distillation process. Due to the substantial memory consumption of depth rendering in Isaac Gym simulator [10], the number of robots trained concurrently is relatively small, leading to higher training costs. Additionally, depth cameras usually operate at lower frame rates, necessitating consideration of camera latency, which significantly increases the training overhead. The student model usually doesn't exceed the performance of the teacher model. The LiDAR-based approach presented by Hoeller et al. [1] is also highly effective and excels in parkour tasks, demonstrating the capability to handle diverse terrains with great proficiency. Due to its goal of enabling robust navigation, this framework is relatively complex, involving multiple modules for perception, navigation, and locomotion, each trained independently. The perception module leverages transformer networks for LiDAR point cloud processing, the navigation module formulates strategies based on the perception data, and the locomotion module executes strategies for tasks such as climbing, jumping, and crouching.

To address the aforementioned issues and to enhance the terrain traversal abilities of quadrupedal robots, we propose SF-TIM, a simple control framework for terrain imagination and measurement. Compared with depth-camera frameworks, our approach does not require distillation, significantly reducing training time. This reduction is due to our ability to directly utilize elevation maps during real-world deployment. Additionally, our elevation maps operate at a frequency of 200 Hz, minimizing errors introduced by latency. Our framework enables a single network to achieve various maneuvers, including climbing upwards, jumping downwards, horizontal jumping, ascending and descending stairs, and controlling locomotion on relatively flat terrain. For jumping maneuvers, we align the robot's heading velocity to the terrain's direction of traversal using remote control and manage its forward speed along the x axis to navigate through the terrain. For other types of terrain, remote commands allow for the adjustment of the robot's velocities in the x and y directions, as well as its angular velocity about the z -axis. We also propose a terrain-guided reward approach specifically to enhance the jumping performance of quadrupedal robots, endowing them to achieve higher terrain levels in simulation. To reduce the sim-to-real gap, we introduce a stable and high-speed elevation map generation framework, facilitating zero-shot sim-to-real transfer of locomotion ability.

In summary, our contributions are as follows:

- · We propose SF-TIM , a robust terrain-guided LiDARbased framework using terrain imagination and measurement.
- · We introduce a terrain-guided reward approach to enhance the jumping performance of quadrupedal robots and develop a stable and high-speed elevation map generation framework to reduce the sim-to-real gap, enabling zero-shot sim-to-real transfer.
- · Our approach simplifies quadrupedal robot training by using a single network trained solely with elevation maps. This network enables effective traversal of stairs and maintains control over horizontal speed and z-axis angular velocity in non-jumping scenarios.

## II. RELATED WORK

This section provides a concise review of notable works related to proprioceptive and exteroceptive sensors quadrupedal robot control frameworks.

A. Learning Quadrupedal Robot Locomotion Using Proprioceptive Sensors Only

This subsection discusses approaches where quadrupedal robots rely exclusively on proprioceptive data. As a result, these robots perceive terrain primarily through contact, using leg or body collisions to detect features such as stairs.

Kumar et al. [11] proposed Rapid Motor Adaptation (RMA), which enables quadruped robots to adapt in real-time to various challenging terrains without prior exposure during training. Wu et al. [12] introduced a locomotion system using Adversarial Motion Priors that enables quadruped robots to traverse challenging terrains robustly and rapidly with only proprioceptive sensors. Long et al. [4] introduced the Hybrid Internal Model (HIM), which leverages the robot's response to disturbances for robust state estimation, enabling efficient learning and agile locomotion across diverse terrains with minimal sensor input. Margolis et al. [13] presented an end-to-end learned controller that achieves record agility for the MIT Mini Cheetah. Zhang et al. [14] proposed a learning approach enabling quadrupedal robots to acquire highly dynamic behaviors such as sprinting, jumping, and sharp turning from animal motion data. Nahrendra et al. [2] proposed DreamWaQ, which uses deep reinforcement learning with implicit terrain imagination to enable quadrupedal robots to traverse challenging terrains with limited sensing modalities. Inspired by DreamWaQ, we introduce terrain imagination into our framework to accelerate agent learning. To further unlock jumping capabilities, we incorporate direct terrain measurement into the framework, integrating it with imagination. This integration allows the agent to adapt to various terrains more effectively, with the aim of improving the locomotion stability.

## B. Learning Quadrupedal Robot Locomotion Using Exteroceptive Sensors

By incorporating exteroceptive sensors such as depth cameras or LiDAR, quadrupedal robots can perceive terrain not only through direct contact or collision but also through advanced sensing capabilities.

Cheng et al. [8] developed an approach for legged robots to perform extreme parkour by initially training a neural network using elevation maps and then employing a teacherstudent method for distillation to operate on depth images from a front-facing camera, enabling precise athletic behaviors despite imprecise actuation and sensing. Zhuang et al. [9] developed an end-to-end vision-based system for quadrupedal robots to autonomously learn and execute diverse parkour skills by training each skill individually and then fusing them into a single policy, enabling navigation of complex environments without reference motion data. Hoeller et al. [1] developed a fully learned approach for agile navigation in quadrupedal robots, combining a highlevel policy that selects and controls locomotion skills with a perception module for reconstructing obstacles from noisy sensory data, enabling the robot to navigate challenging parkour scenarios without expert demonstrations or prior environment knowledge.

Frameworks relying solely on internal proprioceptive sensors have limited capabilities in unlocking the full potential of quadrupedal robot motion. Existing exteroception-based methods also face various challenges, which are relatively difficult, time-consuming, and involve complex systems. Depth map approaches [8], [9] that train with elevation maps

Fig. 2. Overview of the training of SF-TIM. The left side illustrates the actor-critic architecture, which includes the policy network responsible for action selection and the value network responsible for evaluating the expected rewards of states. The right side shows the supervision of the network training process for CENet and the Terrain-feature Encoder.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-e3462d1d-e131-495a-a72f-cc67feb5da89.jpg)

and then switch to depth maps during training are costly and must account for depth map latency, further increasing training costs. Methods using LiDAR sensors [1] involve training multiple skills and adding a navigation module for policy switching, which heavily relies on the robustness of the navigation module and receives external inputs such as global position and time command, resulting in a relatively complex system. To address these issues, we propose a direct training method using elevation maps, which is more costeffective and enables a single network to train for vertical and horizontal jumping capabilities. This approach enables control of the robot's velocities in the x and y directions and its angular velocity about the z axis, while also preserving the ability to handle stairs, without requiring distillation or policy switching.

## III. SF-TIM: PROPOSED FRAMEWORK

SF-TIM aims to enhance the agility and jumping performance of quadrupedal robots by integrating terrain measurement with imaginative processing within a unified network.

## A. Original Inputs of the Agent

Original inputs of the agent include observation o t , observation history o H t , and elevation map e t . The observation vector at time t is defined as:

$$o _ { t } = \left [ \omega _ { t } \ g _ { t } \ c _ { t } \ \theta _ { t } \ \dot { \theta } _ { t } \ a _ { t - 1 } \right ] ^ { T }. \quad \ ( 1 )$$

Here, ω t represents the body angular velocity, g t is the gravity vector in the body frame, c t denotes the body velocity command, θ t corresponds to the joint angle, ˙ θ t is the joint angular velocity, and a t -1 indicates the previous action.

We define a temporal observation vector at time t encompassing the past H measurements as:

$$\mathrm o _ { t } ^ { H } = \left [ \mathrm o _ { t } \ \mathrm o _ { t - 1 } \ \dots \ \mathrm o _ { t - H } \right ] ^ { T }. \quad \quad ( 2 ) \quad \text{where}$$

This vector includes past observations to capture historical data for improved decision-making. For our study, we set H = 5 , significantly enhancing data efficiency during training. This approach allows the policy to explore a wide range of trajectories during training, thereby increasing its robustness through generalization.

The elevation map e t is an exteroceptive input representing a scan of the robot's environment.

## B. Actor and Critic

To enhance quadrupedal robot jumping agility, we employ an asymmetric actor-critic architecture [15], considering that the interplay between the policy and value networks in actorcritic algorithms is sufficient to develop a robust locomotion policy. This architecture is capable of implicitly inferring privileged observations from partial temporal observations and elevation maps, as depicted in Fig. 2.

The policy network, represented as π ϕ ( a t | o t , v t , z p t , z e t ) , is a neural network parameterized by ϕ . This network determines an action a t based on the proprioceptive observation o t , body velocity v t , proprioceptive sensor latent state z p t , and exteroceptive sensor latent state z e t . The policy is optimized using the Proximal Policy Optimization (PPO) algorithm [16].

The action space is represented by a 12 -dimensional vector, a t , which corresponds to the desired joint angles of the robot. To streamline the learning process, the policy is trained to predict the desired joint angles relative to the robot's default standing pose, θ stand . Therefore, the desired joint angles are given by the following equation:

$$\theta _ { \text{des} } = \theta _ { \text{stand} } + a _ { t }.$$

These desired joint angles are then tracked using a Proportional-Derivative (PD) controller for each joint.

The value network is structured to provide an estimation of the state value, V ( s t ) . In contrast to the policy network, the value network receives a privileged observation, s t , defined as follows:

$$\Phi _ { 0 } ^ { \Phi } = \left [ o _ { t } \text{ } v _ { t } \text{ } d _ { t } \text{ } e _ { t } \right ] ^ { T },$$

where d t is the disturbance force applied arbitrarily to the robot's body, and e t is the elevation map scan of the robot's environment, acting as an exteroceptive input. Within the SFTIM framework, the policy network is trained to implicitly infer e t using proprioceptive data.

## C. Combination of Terrain Imagination and Measurement

The context-aided estimator network (CENet) is used to transform o H t into body velocity ˜ v t and the proprioceptive sensor latent state z p t . The terrain-feature encoder transforms e t into the terrain-feature vector z e t .

Inspired by DreamWaQ [2], we employ a context vector z p t , which encapsulates a latent representation of the world state. This context vector facilitates the integration of temporal and observational data, enhancing the robustness and adaptability of our approach. However, context vector z p t usually only reflects the terrain information around the robot, especially the area under the feet, but this information usually cannot enhance the robot's jumping agility. Therefore, we introduce terrain-feature vector z e t to allow the policy to stimulate the potential of jumping.

CENet is capable of estimating both the robot's forward and backward dynamics as well as a latent representation of the environment. It employs a single encoder and a multi-head decoder architecture, as illustrated in the topright corner of Fig. 2. The encoder network transforms o H t into ˜ v t and latent z p t . The first decoder head estimates ˜ v t , while the second head reconstructs o t +1 . We utilize a β -variational auto-encoder ( β -VAE) [17]-[19] framework for the auto-encoder setup.

The optimization of CENet involves a hybrid loss function:

$$\mathcal { L } _ { \text{CE} } = \mathcal { L } _ { \text{est} } + \mathcal { L } _ { \text{VAE} }, \text{ \quad \ \ } ( 5 ) \text{ \ \ } \text{dire}$$

where L est and L VAE represent the losses for body velocity estimation and VAE, respectively. The body velocity estimation loss, L est , is defined using Mean Squared Error (MSE):

$$\mathcal { L } _ { \text{est} } = MSE ( \tilde { \mathbf v } _ { t }, \mathbf v _ { t } ),$$

where ˜ v t is the estimated body velocity and v t is the ground truth from the simulator. The VAE loss, L VAE, is formulated as:

$$\mathcal { L } _ { \text{VAE} } = \text{MSE} ( \tilde { \mathfrak { o } } _ { t + 1 }, \mathfrak { o } _ { t + 1 } ) + \beta D _ { \text{KL} } ( q ( \mathbf z _ { t } ^ { p } | \mathfrak { o } _ { t } ^ { H } ) \ \| \ p ( \mathbf z _ { t } ^ { p } ) ), \ ( 7 )$$

where ˜ o t +1 is the reconstructed next observation, q ( z p t | o H t ) is the posterior distribution of z p t given o H t , and p ( z p t ) is the prior distribution (a standard normal distribution in this case). The reconstruction loss is computed using MSE, while the KL divergence serves as the latent loss in the VAE training process. This approach ensures effective encoding of o H t into meaningful latent representations ˜ v t and z p t , thereby enhancing the robustness of CENet for state estimation tasks. Since CENet generally relies on past historical states to obtain the current terrain's implicit representation or to predict future terrain, such as in stair scenarios, it cannot effectively predict jumping scenarios based on past information.

To address the aforementioned issue, we incorporate terrain measurement observations and use a terrain-feature encoder to extract terrain features. The terrain-feature Encoder network, as shown in the bottom-right corner of Fig. 2, transforms e H t into terrain-feature latent z p t and the Terrainfeature Encoder network transforms z p t into ˜ e H t . The elevation map reconstruction loss is also defined using MSE:

$$\mathcal { L } _ { \text{terrain} } = \text{MSE} ( \tilde { e } _ { t } ^ { H }, \text{e} _ { t } ^ { H } ), \quad \quad ( 8 ) \quad \tau _ { 5 }, \colon$$

where ˜ e H t is the estimated elevation map and e H t is the ground truth from the simulator.

## D. Reward Function

Given the goal of enhancing the jumping performance of the quadruped robot, certain reward functions have been refined to address various terrain categories, such as omitting penalties for the robot's y-axis angular velocity and pitch angle. It comprises task rewards for tracking the commanded velocity and stability rewards to ensure stable and natural locomotion behavior. The specifics of the reward function are detailed in Table I. The total reward for the policy, given an action at each state, is formulated as:

$$r _ { t } ( s _ { t }, \mathbf a _ { t } ) = \sum _ { i } r _ { i } w _ { i },$$

where i indexes each reward component listed in Table I, with the rewards for feet edge and feet stumble referring to previous work [8].

Due to limitations in the x-speed tracking command of the quadruped robot, it struggles with upward jumping on τ 5 terrain. To address this, we develop a tailored reward using terrain-specific linear speed tracking. This method utilizes terrain information to orientate the quadruped robot's speed direction. As a result, when approaching a platform, the expected speed direction aligns not with the robot's xdirection, but rather with the direction necessary to cross the platform effectively. First, we select the elevation map point set P of the 1 6 . m × 1 0 . m area near the fuselage to fit the plane, where the normal vector of the plane is ˆ n t . Then we obtain the direction ˆ n v through the terrain.

$$\begin{matrix} ( 6 ) & & \\ \text{ground} & & R _ { t } = \begin{bmatrix} \succeq \\ \cos ( - \arcsin ( \hat { n } _ { t } ( 0 ) ) ) & 0 & \sin ( - \arcsin ( \hat { n } _ { t } ( 0 ) ) ) \\ 0 & & 1 & 0 \\ - \sin ( - \arcsin ( \hat { n } _ { t } ( 0 ) ) ) & 0 & \cos ( - \arcsin ( \hat { n } _ { t } ( 0 ) ) ) \\ & & & & & ( 1 0 ) \\ \end{bmatrix}, \\ \dots & \dots & & \coprod 1 \end{matrix}$$

$$\hat { n } _ { v } = R _ { t } \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}.$$

Here, R t is the rotation matrix calculated with roll Φ = 0 , pitch Θ = -arcsin(ˆ (0)) n t , and yaw Ψ = 0 . We design a terrain-guided linear velocity tracking (T-L-tracking) reward function, min( &lt; v world , ˆ n v &gt;,v cmd x ) , which encourages the velocity direction to align with ˆ n v as the quadruped approaches the edge of a high platform.

## E. Curriculum Learning

We employ simulation-based training methodologies at the Isaac gym facility [10]. Our approach incorporates a gameinspired curriculum [20], which facilitates the incremental acquisition of locomotion policies adept at traversing complex terrains. This progressive learning paradigm enhances the robustness and adaptability of the developed locomotion strategies. To enable the robot to perform vertical and horizontal movements, as well as navigate stairs and cross flat or small obstacles, we utilize five types of terrain: slopes τ 1 , discrete stones τ 2 , staircases τ 3 , gaps τ 4 , and high platforms τ 5 , as shown in Fig. 4. The τ 4 and τ 5 require only forward

Fig. 3. Overview of the real-world deployment of SF-TIM. The yellow section in the lower-left corner represents the elevation map acquisition module e t , and the right side shows the forward propagation inference of the network.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-c693de16-ae06-403c-ac92-b6105cfacd9f.jpg)

TABLE I

TABLE II

REWARD FUNCTIONS AND THEIR RESPECTIVE WEIGHTS.

Fig. 4. Different terrain types. Slopes: τ 1 , Discrete stone: τ 2 , Staircases: τ 3 , Gaps: τ 4 , High platforms: τ 5 .

| Reward                   | Equation ( r i )                        | Weight ( w i )   |
|--------------------------|-----------------------------------------|------------------|
| T-L-tracking( τ 5 )      | min( < v world , ˆ n v >,v cmd x ) 2    | 3.0              |
| L-tracking( τ 1 ∼ τ 4 )  | 2 e - 4( v cmd xy - v xy )              | 3.0              |
| A-tracking( τ 1 ∼ τ 5 )  | 0 . 5 exp( - 4( ω cmd yaw - ω yaw ) 2 ) | 0.5              |
| v z ( τ 1 ∼ τ 4 )        | - v 2 z                                 | -2.0             |
| ω x ( τ 1 ∼ τ 5 )        | - ω 2 x                                 | -0.05            |
| Roll( τ 1 ∼ τ 5 )        | -| g (0) - ˆ n t (0) | 2                | -10.0            |
| Yaw( τ 5 )               | - yaw 2                                 | -1.0             |
| Joint acc( τ 1 ∼ τ 5 )   | - ˙ θ 2                                 | - 2 . 5 × 10 - 7 |
| Body height( τ 1 ∼ τ 5 ) | - ( h des - h ) 2                       | -10.0            |
| Action rate( τ 1 ∼ τ 5 ) | - ( a t - a t - 1 ) 2                   | -0.04            |
| Smoothness( τ 1 ∼ τ 5 )  | - ( a t - 2 a t - 1 + a t - 2 ) 2       | -0.03            |
| Hip angle( τ 1 ∼ τ 5 )   | - ( d hip des - d hip ) 2               | -1.0             |
| Feet edge( τ 4 )         |                                         | -10.0            |
| Feet edge( τ 5 )         |                                         | -1.0             |
| Feet stumble( τ 4 )      |                                         | -10.0            |
| Feet stumble( τ 5 )      |                                         | -1.0             |

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-dda34b78-6da7-454c-b36d-69b6e0a59adf.jpg)

speed commands, whereas the other terrains allow for both forward and lateral speed commands, as well as angular rotation around the z-axis shown in Table II.

## F. Elevation Map Generation

To minimize the sim-to-real gap during the real-world deployment phase and improve the performance of quadrupedal robots in jumping tasks, we propose a low-latency and highquality elevation map generation module suitable for SFTIM, as shown in the lower left corner of the real-world deployment overflow in Fig. 3.

DIFFERENT COMMAND RANGE IN DIFFERENT TERRAIN.

| Command            | Terrain type   | Range              |
|--------------------|----------------|--------------------|
| Linear velocity x  | τ 1 ∼ τ 3      | [ - 1 . 2 , 1 . 2] |
| Linear velocity x  | τ 4 ∼ τ 5      | [0 . 3 , 1 . 2]    |
| Linear velocity y  | τ 1 ∼ τ 3      | [ - 1 . 2 , 1 . 2] |
| Linear velocity y  | τ 4 ∼ τ 5      | 0                  |
| Angular velocity z | τ 1 ∼ τ 3      | [ - 2 . 0 , 2 . 0] |
| Angular velocity z | τ 4 ∼ τ 5      | 0                  |

We utilize Fast-LIO2 [21] to obtain undistorted point clouds and LiDAR odometry at 10 Hz, using point cloud and IMU data as inputs. For instance, using 10 Hz LiDAR odometry at a speed of 1 m/s can result in an error of approximately 10 cm, which can affect the timing of jumps. To obtain higher frequency odometry, we employ a quaternion-based Extended Kalman Filter (EKF) to fuse the LiDAR odometry with IMU data, resulting in 200 Hz odometry output. The distortion-corrected point clouds are then fed into the elevation map management module, which maintains global elevation map information. Simultaneously, this module utilizes high-frequency odometry data to generate elevation maps of the quadruped robot's surroundings. Consequently, the acquisition of high-frequency elevation maps reduces the sim-to-real gap.

The editability of the elevation map module can bring more flexibility. For example, we can generate virtual deep trenches in the world coordinate system, so that the robot can jump even on flat ground as shown in Fig. 1 (a) . This method not only provides flexibility during debugging but also allows the robot to jump according to user needs, such as visual semantic detection, which enables the robot to jump actively to avoid puddles or more dangerous scenes that cannot be detected by using depth maps.

## IV. EXPERIMENTS

We utilize the Isaac Gym simulator, built upon the opensource framework outlined in [20], to concurrently train the policy, value, CENet networks [2] and terrain-feature encoder and decoder networks. The training is conducted in parallel with 4 096 , agents subjected to domain randomization. Domain randomization is employed to enhance the robustness

TABLE III

QUALITATIVE COMPARISON BETWEEN SF-TIM AND OTHER QUADRUPEDAL ROBOT LEARNING ALGORITHMS.

| Method             | Exteroceptive sensor   | No individual skill training   | Use one policy   | Control v y   | Control ω z   |
|--------------------|------------------------|--------------------------------|------------------|---------------|---------------|
| Cheng et al. [8]   | Depth camera           | Yes                            | Yes              | ×             | ×             |
| Zhuang et al. [9]  | Depth camera           | No                             | Yes              | ×             | ×             |
| Hoeller et al. [1] | LiDAR                  | No                             | No               | ✓             | ✓             |
| SF-TIM (ours)      | LiDAR                  | Yes                            | Yes              | ✓             | ✓             |

and generalization of the learned policies by varying environmental parameters during training. Table VI details the randomized parameters used. All algorithms employ PPO [16] for training the policy network, with a clipping range of 0 2 . , a generalized advantage estimation factor of 0 95 . , and a discount factor of 0 99 . . The networks are optimized using the Adam optimizer [22] with a learning rate set to 10 -3 . All training is performed on a desktop PC with an Intel Core i7-14700 CPU @ 3.40 GHz, 32 GB RAM, and an NVIDIA RTX 4090Ti GPU.

TABLE IV TERRAIN PARAMETERS OF LITE3 AND X30.

Given the complexity of our task, we approach training in two distinct stages. Initially, we focus on teaching the robot to walk on terrains τ 1 , τ 2 , and τ 3 , refining its walking policy ( P trot ) until it reaches a stable state. Following this foundational training, we transition to a more challenging allterrain ( τ 1 ∼ τ 5 ) regimen. This two-step approach mitigates risks of the robot attempting to jump prematurely on standard terrains ( τ 1 ∼ τ 3 ), such as using a pronking gait, which could occur if proceeding with direct, single-phase training.

## A. Qualitative Comparisons with Other Quadrupedal Robot Learning Algorithms

We quantitatively compare our quadrupedal robot learning algorithms SF-TIM with other known algorithms, as shown in Table III. Two main considerations are made regarding the complexity of the training system. These include whether to train parkour skills separately and whether to use a single policy. Additionally, two functional indicators of the physical system are considered, namely the ability to control lateral velocity and the ability to control angular velocity around the z-axis. Notably, in terms of training system indicators, using a single policy eliminates the need for policy switching. Systems that do require policy switching usually need a robust policy selection module, which significantly impacts system performance. The two functional indicators for the physical system enhance operability after deployment, such as repetitive jumping onto the same platform.

## B. Terrain-guided Reward Simulation Experiment

We set up 10 levels of all terrains ( l ∈ [0 , 9] ). The level table of different terrain parameters is shown in Table IV. Among all terrains, τ 5 is relatively more difficult, so we design a terrain-guided tracking velocity reward function specifically for jumping in τ 5 . For a comparative evaluation, we compare the method with or without our designed terrain guidance reward in the second step of training. In the method without our designed reward, the velocity-tracking reward in τ 5 remains the same as that in τ 1 ∼ τ 4 . Due

TABLE V COMPARISON OF OUR APPROACH WITH AND WITHOUT

| Robot   | Terrain             | Terrain parameter                                                                    | m                                                                                       |
|---------|---------------------|--------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------|
| Lite3   | τ 1 τ 2 τ 3 τ 4 τ 5 | Slope height difference Discrete stone height Stair height Gap width Platform height | 0 . 05 × l 0 . 05 +0 . 025 × l 0 . 05 +0 . 013 × l 0 . 2 +0 . 035 × l 0 . 1 +0 . 05 × l |
| X30     | τ 1 τ 2 τ 3 τ 4 τ 5 | Slope height difference Discrete stone height Stair height Gap width Platform height | 0 . 05 × l 0 . 05 +0 . 035 × l 0 . 05 +0 . 018 × l 0 . 2 +0 . 06 × l 0 . 1 +0 . 07 × l  |

TERRAIN-GUIDED REWARD DESIGNS.

| Method                                         | Robot   | Terrain level   | L6 SR(%)   | L9 SR(%)   |
|------------------------------------------------|---------|-----------------|------------|------------|
| SF-TIM w/ T-L-tracking SF-TIM w/o T-L-tracking | Lite3   | 6.0 4.8         | 98 96      | 95 15      |
| SF-TIM w/ T-L-tracking SF-TIM w/o T-L-tracking | X30     | 6.1 5.5         | 99 98      | 95 80      |

to differences in reward function design, we do not use the reward magnitude for comparison. Instead, we compare the average level of the overall terrain after 1000 iterations, where both networks have converged. Additionally, in τ 5 , we compare the Success Rates (SR) of different Level 6 (L6) and Level 9 (L9) terrains. The table shows that our terrainguided reward function significantly improves the success rate on challenging terrains. For the Lite3 robot, the success rate on level 9 (L9 SR) increased from 15% to 95% with T-L-tracking. Similarly, for the X30 robot, the success rate on level 9 rises from 80% to 95% . These results highlight the importance of terrain-guided rewards in enhancing the performance and robustness of quadrupedal robots.

## C. Real-World Experimental Setup

Real-world experiments were conducted using a Deeprobotics Lite3 robot and an X30 robot. The X30 robot is equipped with four Livox Mid360 LiDARs, while the Lite3 robot is equipped with one Livox Mid360 LiDAR. The Lite3's elevation map generation module and the motion strategy module are run on NVIDIA NX and RK3588 respectively. The two parts of the X30 robot run on two separate RK3588 boards. Communication between the two boards is achieved using User Datagram Protocol (UDP). During inference, the policy operates synchronously with the CENet at 50 Hz. The PD controller tracks the desired joint

Fig. 5. Robustness experiment of the high jump platform of the X30 robot. The X30 robot jumps up and down the platform three times consecutively. On the final lap, it entered a cluttered environment where its calf became entangled with flexible debris but successfully got rid of the clutter.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-8a0bb15a-317b-4701-a643-98c0cd6a128c.jpg)

TABLE VI DOMAIN RANDOMIZATION RANGES APPLIED IN THE SIMULATION.

| Parameter                               | Randomization range   | Unit      |
|-----------------------------------------|-----------------------|-----------|
| Payload                                 | [ - 1 , 2]            | kg        |
| K p factor                              | [0 . 9 , 1 . 1]       | Nm / rad  |
| K d factor                              | [0 . 9 , 1 . 1]       | Nms / rad |
| Motor strength factor                   | [0 . 9 , 1 . 1]       | Nm        |
| Center of mass shift                    | [ - 50 , 50]          | mm        |
| Friction coefficient                    | [0 . 2 , 1 . 25]      | -         |
| System delay                            | [0 . 0 , 15 . 0]      | ms        |
| Noise ratio in elevation map            | [0 . 0 , 0 . 1]       | -         |
| Magnitude of noise in the elevation map | [ - 1 . 0 , 2 . 0]    | m         |

being equipped with only one Livox Mid360 LiDAR, resulting in a smaller perception range compared to the X30, the Lite3 still performed robustly. Additional experimental sites for Lite3 were established, as depicted in Fig. 6. Initially, experiments are conducted on flat ground with one complete circuit around the site, followed by two full traversals of the terrain. The terrain is set up sequentially with a 32 cm high platform, an 8 cm low step, a 40 cm gap, and a threestep staircase with each step being 13 3 . cm high. The Lite3 robot successfully traversed this terrain twice and exhibited agile maneuverability on flat ground. Overall, our algorithm demonstrates strong robustness and performs effectively in traversing diverse terrains.

angles using proportional and derivative gains, with Kp = 28 and Kd = 0 7 . , respectively. For the Lite3 robot, the PD controller gains are Kp = 30 0 . and Kd = 1 0 . , whereas for the X30 robot, they are Kp = 120 0 . and Kd = 3 0 . .

## D. Long-Time Jumping Test and Robustness Analysis

We deploy our algorithm SF-TIM on X30 and Lite3, respectively, conducting repeated experiments in challenging scenarios such as raised platforms and gap crossings to validate the robustness of our algorithm. The experimental results of jumping platforms with X30 are shown in Fig. 5. The X30 robot jumps up and down the platform three times consecutively, with the final landing into a cluttered environment where the quadrupedal robot's calf becomes entangled with flexible debris. Despite this, the algorithm's strong robustness allowed us to clear the debris using remote control. In the experiment, we employ a single policy to control the robot's steering, forward and backward movements, as well as lateral movement.

To verify the universality and robustness of our algorithm across different quadruped robots, we have deployed and conducted experiments on the Lite3 quadruped robot. Despite

## V. CONCLUSIONS AND LIMITATION

In this paper, we present a novel and robust terrain-guided LiDAR parkour framework, denoted as SF-TIM, utilizing elevation maps to address the challenges associated with quadrupedal robot terrain traversal. Compared to existing depth-camera parkour frameworks, our approach significantly reduces training time by training only the teacher network. The frequency of elevation maps is synchronized with the localization frequency, effectively mitigating latency errors. We have successfully demonstrated climbing, jumping, and traversing various terrains, as well as controlling locomotion on flat terrain, all through a single network. The proposed terrain-guided reward approach enhances the jumping performance of quadrupedal robots, facilitating higher terrain level achievements in simulation. Furthermore, the integration of a stable and high-speed elevation map generation framework aims to bridge the sim-to-real gap.

It is essential to acknowledge the limitations of our approach. Our framework mainly focuses on jumping maneuvers and may not cover a wide range of parkour actions. Specifically, our current implementation does not include

Fig. 6. Robustness experiment of the various terrains of the Lite3 robot. After circling flat ground once, Lite3 traversed a 32 cm high platform, an 8 cm low step, a 40 cm gap, and a three-step staircase (each step 13 3 . cm high) twice.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00486v1-figure-37f37cbb-1990-4513-9cc4-f4239705253d.jpg)

actions such as traversing narrow gaps or squeezing through low passages. We intend to increase the number of command channels for remote control to support these actions and further enhance the mobility of quadrupedal robots.

## REFERENCES

- [1] D. Hoeller, N. Rudin, D. Sako, and M. Hutter, 'Anymal parkour: Learning agile navigation for quadrupedal robots,' Science Robotics , vol. 9, no. 88, p. eadi7566, 2024.
- [2] I. M. A. Nahrendra, B. Yu, and H. Myung, 'DreamWaQ: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning,' in 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023, pp. 5078-5084.
- [3] T. He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi, 'Agile but safe: Learning collision-free high-speed legged locomotion,' arXiv preprint arXiv:2401.17583 , 2024.
- [4] J. Long, Z. Wang, Q. Li, L. Cao, J. Gao, and J. Pang, 'Hybrid internal model: Learning agile legged locomotion with simulated robot response,' in International Conference on Learning Representations (ICLR) , 2024.
- [5] J. Ren, Y. Liu, Y. Dai, and G. Wang, 'TOP-Nav: Legged navigation integrating terrain, obstacle and proprioception estimation,' arXiv preprint arXiv:2404.15256 , 2024.
- [6] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, 'Learning robust perceptive locomotion for quadrupedal robots in the wild,' Science Robotics , vol. 7, no. 62, p. eabk2822, 2022.
- [7] L. Xu, K. Chai, Z. Han, H. Liu, C. Xu, Y. Cao, and F. Gao, 'An efficient trajectory planner for car-like robots on uneven terrain,' in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 2853-2860.
- [8] X. Cheng, K. Shi, A. Agarwal, and D. Pathak, 'Extreme parkour with legged robots,' arXiv preprint arXiv:2309.14341 , 2023.
- [9] Z. Zhuang et al. , 'Robot parkour learning,' in Conference on Robot Learning (CoRL) , 2023.
- [10] V. Makoviychuk et al. , 'Isaac gym: High performance GPU-based physics simulation for robot learning,' in Advances in Neural Information Processing Systems (NeurIPS) , 2021.
- [11] A. Kumar, Z. Fu, D. Pathak, and J. Malik, 'RMA: Rapid motor adaptation for legged robots,' in Robotics: Science and Systems (RSS) , 2021.
- [12] J. Wu, G. Xin, C. Qi, and Y. Xue, 'Learning robust and agile legged locomotion using adversarial motion priors,' IEEE Robotics and Automation Letters , vol. 8, no. 8, pp. 4975-4982, 2023.
- [13] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, 'Rapid locomotion via reinforcement learning,' The International Journal of Robotics Research , vol. 43, no. 4, pp. 572-587, 2024.
- [14] C. Zhang et al. , 'Learning highly dynamic behaviors for quadrupedal robots,' arXiv preprint arXiv:2402.13473 , 2024.
- [15] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel, 'Asymmetric actor critic for image-based robot learning,' in Robotics: Science and Systems (RSS) , 2018.
- [16] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, 'Proximal policy optimization algorithms,' arXiv preprint arXiv:1707.06347 , 2017.
- [17] D. P. Kingma and M. Welling, 'Auto-encoding variational Bayes,' in International Conference on Learning Representations (ICLR) , 2014.
- [18] I. Higgins et al. , ' β - VAE: Learning basic visual concepts with a constrained variational framework,' in International Conference on Learning Representations (ICLR) , 2017.
- [19] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, 'Understanding disentangling in β - VAE,' Advances in Neural Information Processing (NeurIPS) Workshop on Learning Disentangled Representations , 2017.
- [20] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, 'Learning to walk in minutes using massively parallel deep reinforcement learning,' in Conference on Robot Learning (CoRL) , 2022, pp. 91-100.
- [21] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang, 'FAST-LIO2: Fast direct LiDAR-inertial odometry,' IEEE Transactions on Robotics , vol. 38, no. 4, pp. 2053-2073, 2022.
- [22] P. K. Diederik, 'Adam: A method for stochastic optimization,' in International Conference on Learning Representations (ICLR) , 2015.
Ze Wang Yang Li Long Xu Hao Shi Zunwang Ma Zhen Chu Chao Li Fei Gao Kailun Yang Kaiwei Wang
SF-TIM: A Simple Framework for Enhancing Quadrupedal Robot Jumping Agility by Combining Terrain Imagination and Measurement
Dynamic jumping on high platforms and over gaps differentiates legged robots
from wheeled counterparts. Compared to walking on rough terrains, dynamic
locomotion on abrupt surfaces requires fusing proprioceptive and exteroceptive
perception for explosive movements. In this paper, we propose SF-TIM (Simple
Framework combining Terrain Imagination and Measurement), a single-policy
method that enhances quadrupedal robot jumping agility, while preserving their
fundamental blind walking capabilities. In addition, we introduce a
terrain-guided reward design specifically to assist quadrupedal robots in high
jumping, improving their performance in this task. To narrow the
simulation-to-reality gap in quadrupedal robot learning, we introduce a stable
and high-speed elevation map generation framework, enabling zero-shot
simulation-to-reality transfer of locomotion ability. Our algorithm has been
deployed and validated on both the small-/large-size quadrupedal robots,
demonstrating its effectiveness in real-world applications: the robot has
successfully traversed various high platforms and gaps, showing the robustness
of our proposed approach. A demo video has been made available at
https://flysoaryun.github.io/SF-TIM.