Body: <p>I'm currently working on a project involving a drone simulation within Unreal Engine, utilizing Microsoft's AirSim plugin to facilitate a realistic environment for drone operation. The core objective of my project is to train the drone using reinforcement learning (RL) techniques to navigate through various terrains and obstacles autonomously.</p>
  <p>However, I've encountered a significant challenge: the drone's movement is jerky and lacks the smoothness expected for practical applications.</p>
  <p>The drone tries to fly through a hallway through holes that move. It is able to do this, but not without being jerky.</p>
  <p>Here are some specifics about my setup and what I've tried so far:
  Environment: Unreal Engine with Microsoft's AirSim plugin.
  RL Algorithm: PPO
  Observation Space: The scene of where its at
  Action Space:
  19 different choices:</p>
  <pre><code>def do_action(self, select_action):
  base_speed = 3.0  # Base speed for movement
  
  # Mapping each action to its corresponding velocity adjustments
  action_effects = {
      0: {'vy': 0, 'vz': 0, 'speed': -base_speed},  # Backward
      1: {'vy': 0, 'vz': 0, 'speed': base_speed},   # Forward
      2: {'vy': -base_speed, 'vz': 0, 'speed': 0},  # Left
      3: {'vy': base_speed, 'vz': 0, 'speed': 0},   # Right
      4: {'vy': 0, 'vz': base_speed, 'speed': 0},   # Up
      5: {'vy': 0, 'vz': -base_speed, 'speed': 0},  # Down
      6: {'vy': 0, 'vz': 0, 'speed': 0},            # Hover
      7: {'vy': 0, 'vz': base_speed, 'speed': base_speed},  # Forward-Up
      8: {'vy': 0, 'vz': -base_speed, 'speed': base_speed},  # Forward-Down
      9: {'vy': -base_speed, 'vz': base_speed, 'speed': base_speed},  # Forward-Left-Up
      10: {'vy': -base_speed, 'vz': -base_speed, 'speed': base_speed}, # Forward-Left-Down
      11: {'vy': base_speed, 'vz': base_speed, 'speed': base_speed},  # Forward-Right-Up
      12: {'vy': base_speed, 'vz': -base_speed, 'speed': base_speed}, # Forward-Right-Down
      13: {'vy': 0, 'vz': base_speed, 'speed': -base_speed},  # Backward-Up
      14: {'vy': 0, 'vz': -base_speed, 'speed': -base_speed}, # Backward-Down
      15: {'vy': -base_speed, 'vz': base_speed, 'speed': -base_speed}, # Backward-Left-Up
      16: {'vy': -base_speed, 'vz': -base_speed, 'speed': -base_speed}, # Backward-Left-Down
      17: {'vy': base_speed, 'vz': base_speed, 'speed': -base_speed}, # Backward-Right-Up
      18: {'vy': base_speed, 'vz': -base_speed, 'speed': -base_speed}, # Backward-Right-Down
  }
  
  if select_action in action_effects:
      action = action_effects[select_action]
      self.drone.moveByVelocityBodyFrameAsync(action['speed'], action['vy'], action['vz'], duration=0.1).join()
  else:
      print(f&quot;Action {select_action} is not defined.&quot;)
  </code></pre>
  <p>Reward Function: Right now the reward just gives points for moving forward, getting to the end, or - points for crashing</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-18T18:21:29.960Z
Comments.Text: Any research papers or suggestions would be very helpful!
Comments.UserDisplayName: Samuel Fipps
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-19T12:20:28.033Z
Comments.Text: I think you could come up with solutions if you better define what "jerky" is - if it's the number of movements, the speed, the change of directions, or all of them. You could for example introduce other penalties (e.g for number of movements, getting close to obstacles, high speed, etc.)
Comments.UserDisplayName: Ciodar
Tags: <reinforcement-learning>
Title: How to Prevent Jerky Movements in a Reinforcement Learning-Based Drone Using Unreal Engine and AirSim?
UserDisplayName: Samuel Fipps
ThreadText: # How to Prevent Jerky Movements in a Reinforcement Learning-Based Drone Using Unreal Engine and AirSim?
  
  (Asked by: username_0 on 2024-02-18)
  
  I'm currently working on a project involving a drone simulation within Unreal Engine, utilizing Microsoft's AirSim plugin to facilitate a realistic environment for drone operation. The core objective of my project is to train the drone using reinforcement learning (RL) techniques to navigate through various terrains and obstacles autonomously.
  
  However, I've encountered a significant challenge: the drone's movement is jerky and lacks the smoothness expected for practical applications.
  
  The drone tries to fly through a hallway through holes that move. It is able to do this, but not without being jerky.
  
  Here are some specifics about my setup and what I've tried so far:
  Environment: Unreal Engine with Microsoft's AirSim plugin.
  RL Algorithm: PPO
  Observation Space: The scene of where its at
  Action Space:
  19 different choices:
  
  ```
  def do_action(self, select_action):
  base_speed = 3.0  # Base speed for movement
  
  # Mapping each action to its corresponding velocity adjustments
  action_effects = {
      0: {'vy': 0, 'vz': 0, 'speed': -base_speed},  # Backward
      1: {'vy': 0, 'vz': 0, 'speed': base_speed},   # Forward
      2: {'vy': -base_speed, 'vz': 0, 'speed': 0},  # Left
      3: {'vy': base_speed, 'vz': 0, 'speed': 0},   # Right
      4: {'vy': 0, 'vz': base_speed, 'speed': 0},   # Up
      5: {'vy': 0, 'vz': -base_speed, 'speed': 0},  # Down
      6: {'vy': 0, 'vz': 0, 'speed': 0},            # Hover
      7: {'vy': 0, 'vz': base_speed, 'speed': base_speed},  # Forward-Up
      8: {'vy': 0, 'vz': -base_speed, 'speed': base_speed},  # Forward-Down
      9: {'vy': -base_speed, 'vz': base_speed, 'speed': base_speed},  # Forward-Left-Up
      10: {'vy': -base_speed, 'vz': -base_speed, 'speed': base_speed}, # Forward-Left-Down
      11: {'vy': base_speed, 'vz': base_speed, 'speed': base_speed},  # Forward-Right-Up
      12: {'vy': base_speed, 'vz': -base_speed, 'speed': base_speed}, # Forward-Right-Down
      13: {'vy': 0, 'vz': base_speed, 'speed': -base_speed},  # Backward-Up
      14: {'vy': 0, 'vz': -base_speed, 'speed': -base_speed}, # Backward-Down
      15: {'vy': -base_speed, 'vz': base_speed, 'speed': -base_speed}, # Backward-Left-Up
      16: {'vy': -base_speed, 'vz': -base_speed, 'speed': -base_speed}, # Backward-Left-Down
      17: {'vy': base_speed, 'vz': base_speed, 'speed': -base_speed}, # Backward-Right-Up
      18: {'vy': base_speed, 'vz': -base_speed, 'speed': -base_speed}, # Backward-Right-Down
  }
  
  if select_action in action_effects:
      action = action_effects[select_action]
      self.drone.moveByVelocityBodyFrameAsync(action['speed'], action['vy'], action['vz'], duration=0.1).join()
  else:
      print(f"Action {select_action} is not defined.")
  
  ```
  
  Reward Function: Right now the reward just gives points for moving forward, getting to the end, or - points for crashing
  
  **Tags:** reinforcement-learning
  
  > **Comments:**
  > **username_0**: Any research papers or suggestions would be very helpful!  
  > **username_1**: I think you could come up with solutions if you better define what "jerky" is - if it's the number of movements, the speed, the change of directions, or all of them. You could for example introduce other penalties (e.g for number of movements, getting close to obstacles, high speed, etc.)  
  
  ---
  
  ## 0 answers
  
  