Answers.Body: <p>Short answer: Yes.</p>
  <p>Think about motor controllers. They are all digital these days, so they work exactly like this anyway, with fixed time steps. This is not specific to RL. You will find this assumption everywhere in control theory, e.g. <a href="https://en.wikipedia.org/wiki/Model_predictive_control#Theory_behind_MPC" rel="nofollow noreferrer">model predictive control</a> (Wikipedia).</p>
  <p>The same extends to inputs. If your system takes an undetermined time to process a video frame, its reaction time (latency) will be variable. As we know from basic control theory, latency (&quot;dead time&quot;) is critical and will determine whether your system oscillates. So we don't want systems with unpredictable latency.</p>
  <p>The keyword here is <a href="https://en.wikipedia.org/wiki/Real-time_computing#Criteria_for_real-time_computing" rel="nofollow noreferrer">real time</a> computation. You usually have a hard deadline when you need to deliver the next cycle's target position to your PID controller. If you miss it, it will have to execute <em>some</em> action anyway. If you provide new values more frequently your effort is wasted.</p>
  <p>Now this is mostly about low-level hardware control. When you get further away, say to trajectory planning, things may get a bit more relaxed, and you can just send, say, a new GPS target whenever you like. The controller will still use fixed time steps.</p>
  <p>For a real-live example, take this paper about <a href="https://www.nature.com/articles/s41586-020-2939-8" rel="nofollow noreferrer">stratospheric balloons</a>. They still used RL with a fixed 3 minute control cycle. Or this one about <a href="https://arxiv.org/abs/2312.09906" rel="nofollow noreferrer">solving a labyrinth</a> with RL talks about a control rate of 55Hz. (It looks like they matched it to the camera FPS.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T07:52:46.047Z
Answers.Text: Thanks for the reply! As you mentioned about the real-time computation, I am wondering for a system with multiple components, say, trajectory planner -> cascaded PID controllers (position, velocity, attitude, rate, etc), how to determine which one should have a higher frequency. Or how does the frequency of each component affect the whole system's performance?
Answers.UserDisplayName: bz2000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-25T07:50:05.807Z
Answers.LastActivityDate: 2024-03-25T08:05:18.717Z
Answers.LastEditDate: 2024-03-25T08:05:18.717Z
Answers.UserDisplayName: maxy
Body: <p>I am new to the RL community and I am working on projects about using RL to control robots like drones to fly in the scenes. I have been using Nvidia's Issac Gym for some time, but I have a question about the Gym-like simulation and training process, i.e., <code>action -&gt; reward -&gt; (simulation and physics stepping) -&gt; next obs</code>. In this stepping fashion, the process is synchronized which means the simulator needs to wait for the latest action to be generated. It is different from e.g., simulations in ROS &amp; Gazebo. In ROS &amp; Gazebo, everything is asynchronous. The simulation in Gazebo has its own stepping frequency, and the controller node can publish its commands in some other frequency. So it is not necessarily one control command for one physics stepping.</p>
  <p>So my question is that is the synchronous simulation pattern (Gym-like) good enough for some low-latency robot platforms like drones? Can someone give me some resources on this topic? Thanks!</p>
  
Tags: <reinforcement-learning><gym><robotics>
Title: RL simulation: Does the Gym-like RL training solution fits the real-world environments?
UserDisplayName: bz2000
ThreadText: # RL simulation: Does the Gym-like RL training solution fits the real-world environments?
  
  (Asked by: username_0 on 2024-03-25)
  
  I am new to the RL community and I am working on projects about using RL to control robots like drones to fly in the scenes. I have been using Nvidia's Issac Gym for some time, but I have a question about the Gym-like simulation and training process, i.e., `action -> reward -> (simulation and physics stepping) -> next obs`. In this stepping fashion, the process is synchronized which means the simulator needs to wait for the latest action to be generated. It is different from e.g., simulations in ROS & Gazebo. In ROS & Gazebo, everything is asynchronous. The simulation in Gazebo has its own stepping frequency, and the controller node can publish its commands in some other frequency. So it is not necessarily one control command for one physics stepping.
  
  So my question is that is the synchronous simulation pattern (Gym-like) good enough for some low-latency robot platforms like drones? Can someone give me some resources on this topic? Thanks!
  
  **Tags:** reinforcement-learning, gym, robotics
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-25. Score: 1
  
  Short answer: Yes.
  
  Think about motor controllers. They are all digital these days, so they work exactly like this anyway, with fixed time steps. This is not specific to RL. You will find this assumption everywhere in control theory, e.g. [model predictive control](https://en.wikipedia.org/wiki/Model_predictive_control#Theory_behind_MPC) (Wikipedia).
  
  The same extends to inputs. If your system takes an undetermined time to process a video frame, its reaction time (latency) will be variable. As we know from basic control theory, latency ("dead time") is critical and will determine whether your system oscillates. So we don't want systems with unpredictable latency.
  
  The keyword here is [real time](https://en.wikipedia.org/wiki/Real-time_computing#Criteria_for_real-time_computing) computation. You usually have a hard deadline when you need to deliver the next cycle's target position to your PID controller. If you miss it, it will have to execute *some* action anyway. If you provide new values more frequently your effort is wasted.
  
  Now this is mostly about low-level hardware control. When you get further away, say to trajectory planning, things may get a bit more relaxed, and you can just send, say, a new GPS target whenever you like. The controller will still use fixed time steps.
  
  For a real-live example, take this paper about [stratospheric balloons](https://www.nature.com/articles/s41586-020-2939-8). They still used RL with a fixed 3 minute control cycle. Or this one about [solving a labyrinth](https://arxiv.org/abs/2312.09906) with RL talks about a control rate of 55Hz. (It looks like they matched it to the camera FPS.)
  
  > **Comments:**
  > **username_0**: Thanks for the reply! As you mentioned about the real-time computation, I am wondering for a system with multiple components, say, trajectory planner -> cascaded PID controllers (position, velocity, attitude, rate, etc), how to determine which one should have a higher frequency. Or how does the frequency of each component affect the whole system's performance?  
  
  ---
  
  