Answers.Body: <p>Actually, in most of these algorithms, that state is just used as input for some functions (e.g. some value or policy functions).
  Given the correct class of functions (e.g. recurrent neural networks), these algorithms do have the ability to handle dynamic size observations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-26T22:13:31.133Z
Answers.LastActivityDate: 2023-06-26T22:13:31.133Z
Answers.UserDisplayName: Broele
Body: <p>I have an enviroment with continuous actions and state variables. Every time I reset my env, between 2 and 5 balls spawn randomly in a box of 100x100 size. One of those balls (the red one) will receive an action (direction of movement) and will move according to some physics. This ball will always spawn.</p>
  <p>Notice that the observation space only changes when we reset and start a new episode (when we move a step in time the observation space size remains the same). An example of my env:</p>
  <p><a href="https://i.sstatic.net/zz55U.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zz55U.png" alt="enter image description here" /></a></p>
  <ul>
  <li><strong>Action space</strong>: the angle that we want to move the red ball</li>
  <li><strong>Observation space</strong>: the coordinates of the blue balls</li>
  </ul>
  <p>The problem is that sometimes there will be only one blue ball, or two, or three... so the number of 2D coordinates in the observation will change.</p>
  <p>Most of the reinforcement learning techniques have a fixed size state spaces, how can I deal with my dynamic size observation space and apply algorithms such as AC2, PPO, ... ? Should I consider an observation space with the maximum number of balls and make 0 the coordinates of the ones that didn't spawn ?</p>
  
Tags: <reinforcement-learning><state-spaces><continuous-state-spaces><observation-spaces>
Title: Variable observation space at each episode
UserDisplayName: Optical_flow_lover
ThreadText: # Variable observation space at each episode
  
  (Asked by: username_0 on 2023-06-16)
  
  I have an enviroment with continuous actions and state variables. Every time I reset my env, between 2 and 5 balls spawn randomly in a box of 100x100 size. One of those balls (the red one) will receive an action (direction of movement) and will move according to some physics. This ball will always spawn.
  
  Notice that the observation space only changes when we reset and start a new episode (when we move a step in time the observation space size remains the same). An example of my env:
  
  [![enter image description here](https://i.sstatic.net/zz55U.png)](https://i.sstatic.net/zz55U.png)
  
  * **Action space**: the angle that we want to move the red ball
  * **Observation space**: the coordinates of the blue balls
  
  The problem is that sometimes there will be only one blue ball, or two, or three... so the number of 2D coordinates in the observation will change.
  
  Most of the reinforcement learning techniques have a fixed size state spaces, how can I deal with my dynamic size observation space and apply algorithms such as AC2, PPO, ... ? Should I consider an observation space with the maximum number of balls and make 0 the coordinates of the ones that didn't spawn ?
  
  **Tags:** reinforcement-learning, state-spaces, continuous-state-spaces, observation-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-26. Score: 1
  
  Actually, in most of these algorithms, that state is just used as input for some functions (e.g. some value or policy functions).
  Given the correct class of functions (e.g. recurrent neural networks), these algorithms do have the ability to handle dynamic size observations.
  
  
  ---
  
  