Answers.Body: <p>Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem like a complete description that you could use to predict the dynamics of the combined system and consequences of simple actions (presumably your actions will be torque on either A or B in order to maintain balance). However, I think it can be simplified without losing any information for a value or policy function.</p>
  <p>Assuming the discs always remain in contact, I would recommend you don't track B's position in X,Y - its angular position with respect to A's centre, and A's X position define everything you need to know.</p>
  <p>Asuming you have a simple, perfect friction version with no slipping, but constraints on position of A (similar to cartpole) then I think you have four variables:</p>
  <ol>
  <li>Angular velocity of A (<span class="math-container">$\omega_a$</span> rotation speed)</li>
  <li>Angular velocity of B (<span class="math-container">$\omega_b$</span> rotation speed)</li>
  <li>Angular position of B's contact with A (<span class="math-container">$\theta_c$</span> rotation offset, measured from centre of A)*</li>
  <li>Angular position of A with respect to a start position (<span class="math-container">$\theta_a$</span> rotation offset) <strong>OR</strong> position of A's centre with respect to a start position (<span class="math-container">$x_a$</span> distance offset). These are in a simple ratio <span class="math-container">$x_a = r_a\theta_a$</span> where <span class="math-container">$r_a$</span> is A's radius, so you can use either</li>
  </ol>
  <p>You can drop this to 3 variables if there is no constraint on the position of disc A, because its position is not required to predict the dynamics for balancing B on top of A.</p>
  <p>The physics details you added to the question don't appear to change any of the above. If you want to train an agent so that it can generalise to different values in the same trained model, then you would add the params you want to vary to the state. Plus you would need to train with different values so the approximator can learn how they impact action choices.</p>
  <hr />
  <p>There is another possibility, that I have assumed you are not considering. If A and B's surfaces are allowed to slip - i.e. there is not perfect friction between them, then you potentially have another free variable in that B's motion around the surface of A is not simply determined by combination of A and B's angular velocity. Whether or not there is another variable needed will depend on how you calculate this slippage. Simple friction models will not need another variable, but more complex ones could require state to track whether slip is occurring.</p>
  <p>* I put a <span class="math-container">$c$</span> suffix on this to differentiate it from the natural reading of <span class="math-container">$\theta_b$</span> which would be B's rotation offset around its own centre.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T18:05:48.433Z
Answers.Text: Hi Neil! Thanks for your answer (again)! Reading your answer, I found that I missed to include crucial information about the environment and I will edit my question to include them. In fact, both disc have different weight and size (as on the picture) and physics is also enabled. "A" has a weight of 1.5 Kg and "B" 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment. I guess all the information I just gave modify a lot the observable state... ? Also why "A" position is necessary ? What if -inf < X < inf ?
Answers.UserDisplayName: CyDevos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T20:06:24.190Z
Answers.Text: If there are no constraints on A position, then it is not necessary state for the RL agent to predict value, so you could drop it. I don't think any of the environment parameters that you commented impact the RL state *unless* you want to train the agent so that it can operate in variations of the environment without re-training. If you train and operate the agent with the same parameters, then those params should not change anything. I don't see any physics that would add more degrees of freedom in the state, but that *is* possible, and I mentioned friction and slipping models as possibility.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T20:13:23.420Z
Answers.Text: Ok perfect, thanks for all the info. It really helped :) It's a shame that we cannot DM. Your knowledge on RL would be much appreciated on my project! Thanks again
Answers.UserDisplayName: CyDevos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T20:20:24.310Z
Answers.Text: @CyDevos I'm always happy to answer questions on the site, but I cannot realistically make time to DM and consult on projects, or commit to support a project over time. Here, I think some physics knowledge combined with RL is required to spot the differences between your simulator state and RL state, but lots of experts on this site should have both.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T20:23:48.123Z
Answers.Text: Yeah, I know, thanks for your time and devotion!
Answers.UserDisplayName: CyDevos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-17T07:52:43.483Z
Answers.Text: Hi Neil, I have one more question. I have put a reward of +1 when the angle is > -15 degree and < 15 degree. This reward only apply if both disc are touching. Do you think it makes sense ?
Answers.UserDisplayName: CyDevos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-17T08:33:33.603Z
Answers.Text: @CyDevos Sure that makes sense for the balancing task, effectively the return is how long the agent can maintain balance. By the way, I assumed in my answer that an episode would end if the discs stopped touching - similar to cartpole, the game would give up if you've obviously lost control. That's a guess on my part - if I'm wrong about that, the answer could also be wrong
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-17T09:27:14.167Z
Answers.Text: thanks! The game end if “B” fall on the ground and a bad reward is given
Answers.UserDisplayName: CyDevos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-23T10:15:42.023Z
Answers.Text: as promise, just to let you know that I have successfully make my ball A keep the ball B on top of itself!
Answers.UserDisplayName: CyDevos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-16T15:50:30.417Z
Answers.LastActivityDate: 2023-09-16T20:15:04.307Z
Answers.LastEditDate: 2023-09-16T20:15:04.307Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm making a new environment where I have two sphere (one above the other) in a 2D plan.
  <a href="https://i.sstatic.net/Javi9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Javi9.png" alt="enter image description here" /></a></p>
  <p>I would like some advice on what observation state I should give to my RL.
  Today I have given the following:</p>
  <ol>
  <li>Position of A (X)</li>
  <li>Angular velocity of A</li>
  <li>Position of B (X and Y)</li>
  <li>Angular velocity of B</li>
  <li>Angle between A and B (relative to A)</li>
  </ol>
  <p>This gives in total 6 different state. The goal is to keep the sphere B as long as possible above A.
  I have thought a lot about the observation state and I don't know if the position of B should be relative to the position of A ? or should it be relative to the world ?</p>
  <p>Should I remove or add other state ? WDYT ?</p>
  <h2><strong>Edit</strong></h2>
  <p>I forgot to mentioned crucial information on my environment:
  Both disc have different weight and size (as on the picture) and physics is also enabled. &quot;A&quot; has a weight of 1.5 Kg and &quot;B&quot; 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment.</p>
  
Tags: <reinforcement-learning><deep-rl><environment><gym><state-spaces>
Title: What kind of observation state would you give for that environment?
UserDisplayName: CyDevos
ThreadText: # What kind of observation state would you give for that environment?
  
  (Asked by: username_0 on 2023-09-16)
  
  I'm making a new environment where I have two sphere (one above the other) in a 2D plan.
  [![enter image description here](https://i.sstatic.net/Javi9.png)](https://i.sstatic.net/Javi9.png)
  
  I would like some advice on what observation state I should give to my RL.
  Today I have given the following:
  
  1. Position of A (X)
  2. Angular velocity of A
  3. Position of B (X and Y)
  4. Angular velocity of B
  5. Angle between A and B (relative to A)
  
  This gives in total 6 different state. The goal is to keep the sphere B as long as possible above A.
  I have thought a lot about the observation state and I don't know if the position of B should be relative to the position of A ? or should it be relative to the world ?
  
  Should I remove or add other state ? WDYT ?
  
  ## **Edit**
  
  I forgot to mentioned crucial information on my environment:
  Both disc have different weight and size (as on the picture) and physics is also enabled. "A" has a weight of 1.5 Kg and "B" 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment.
  
  **Tags:** reinforcement-learning, deep-rl, environment, gym, state-spaces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-16. Score: 3 | Accepted answer
  
  Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem like a complete description that you could use to predict the dynamics of the combined system and consequences of simple actions (presumably your actions will be torque on either A or B in order to maintain balance). However, I think it can be simplified without losing any information for a value or policy function.
  
  Assuming the discs always remain in contact, I would recommend you don't track B's position in X,Y - its angular position with respect to A's centre, and A's X position define everything you need to know.
  
  Asuming you have a simple, perfect friction version with no slipping, but constraints on position of A (similar to cartpole) then I think you have four variables:
  
  1. Angular velocity of A ($\omega\_a$ rotation speed)
  2. Angular velocity of B ($\omega\_b$ rotation speed)
  3. Angular position of B's contact with A ($\theta\_c$ rotation offset, measured from centre of A)\*
  4. Angular position of A with respect to a start position ($\theta\_a$ rotation offset) **OR** position of A's centre with respect to a start position ($x\_a$ distance offset). These are in a simple ratio $x\_a = r\_a\theta\_a$ where $r\_a$ is A's radius, so you can use either
  
  You can drop this to 3 variables if there is no constraint on the position of disc A, because its position is not required to predict the dynamics for balancing B on top of A.
  
  The physics details you added to the question don't appear to change any of the above. If you want to train an agent so that it can generalise to different values in the same trained model, then you would add the params you want to vary to the state. Plus you would need to train with different values so the approximator can learn how they impact action choices.
  
  ---
  
  There is another possibility, that I have assumed you are not considering. If A and B's surfaces are allowed to slip - i.e. there is not perfect friction between them, then you potentially have another free variable in that B's motion around the surface of A is not simply determined by combination of A and B's angular velocity. Whether or not there is another variable needed will depend on how you calculate this slippage. Simple friction models will not need another variable, but more complex ones could require state to track whether slip is occurring.
  
  \* I put a $c$ suffix on this to differentiate it from the natural reading of $\theta\_b$ which would be B's rotation offset around its own centre.
  
  > **Comments:**
  > **username_0**: Hi Neil! Thanks for your answer (again)! Reading your answer, I found that I missed to include crucial information about the environment and I will edit my question to include them. In fact, both disc have different weight and size (as on the picture) and physics is also enabled. "A" has a weight of 1.5 Kg and "B" 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment. I guess all the information I just gave modify a lot the observable state... ? Also why "A" position is necessary ? What if -inf < X < inf ?  
  > **username_1**: If there are no constraints on A position, then it is not necessary state for the RL agent to predict value, so you could drop it. I don't think any of the environment parameters that you commented impact the RL state *unless* you want to train the agent so that it can operate in variations of the environment without re-training. If you train and operate the agent with the same parameters, then those params should not change anything. I don't see any physics that would add more degrees of freedom in the state, but that *is* possible, and I mentioned friction and slipping models as possibility.  
  > **username_0**: Ok perfect, thanks for all the info. It really helped :) It's a shame that we cannot DM. Your knowledge on RL would be much appreciated on my project! Thanks again  
  > **username_1**: @CyDevos I'm always happy to answer questions on the site, but I cannot realistically make time to DM and consult on projects, or commit to support a project over time. Here, I think some physics knowledge combined with RL is required to spot the differences between your simulator state and RL state, but lots of experts on this site should have both.  
  > **username_0**: Yeah, I know, thanks for your time and devotion!  
  > **username_0**: Hi Neil, I have one more question. I have put a reward of +1 when the angle is > -15 degree and < 15 degree. This reward only apply if both disc are touching. Do you think it makes sense ?  
  > **username_1**: @CyDevos Sure that makes sense for the balancing task, effectively the return is how long the agent can maintain balance. By the way, I assumed in my answer that an episode would end if the discs stopped touching - similar to cartpole, the game would give up if you've obviously lost control. That's a guess on my part - if I'm wrong about that, the answer could also be wrong  
  > **username_0**: thanks! The game end if “B” fall on the ground and a bad reward is given  
  > **username_0**: as promise, just to let you know that I have successfully make my ball A keep the ball B on top of itself!  
  
  ---
  
  