Answers.Body: <p>To turn this delayed action approach into a normal, and theoretically valid MDP, add a &quot;pending action resolution&quot; array to the state representation, and ensure that the state transition (or <code>step</code> code) manages this array with each new action pushed into it, and the array shifting down as it goes. The array length should be minimum of the number of timesteps required to fully resolve a previous action.</p>
  <p>This allows your agent to see the actions it has recently performed and account for them in calculations without any adjustment needed to the core MDP formulation or standard agent designs.</p>
  <p>If delays are stochastic or vary (e.g. a scenario where a warehouse is ordering stock), then add more data to this pending action resolutions array as necessary, such as expected resolution time, or time since action was taken.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-03T20:40:02.417Z
Answers.Text: From what I understand in implementation of your 'pending action resolution' array is:  First-In the step(action), take an action in real life. Second- Buffer/queue the action into the array with length of the delay step. Third - repeat fist&second until the action resolution array is completely filled. Last - With the full array, measure the state + reward associated with the oldest/last index of the pending action resolution array. Is this correct? If so, what do I return at the end of the step() function in the Second if I do not have the next state observed and the reward?
Answers.UserDisplayName: Ykwk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-03T20:50:31.497Z
Answers.Text: @Ykwk: That's overcomplicated. The array of recent actions becomes just a normal part of the state. Initialise it with "null" actions so it has a fixed length. When the agent takes an action, push the id of that action onto the array, then shift the array to remove one old action (so the array stays the same length, and actions are stored in age order). Run all the rest of MDP and time step handling as normal for the agent - i.e. process value estimates, use replay memory for training etc on every time step.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T03:32:05.163Z
Answers.Text: How can the 'initialization null' array of action be implemented, when every iteration of step() requires to return -next state -reward? Passing in 'null' action does not give me any return values for step() function. I don't think I can get the next action, without having the agent to go to next state reward.
Answers.UserDisplayName: Ykwk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T06:16:04.627Z
Answers.Text: @Ykwk These are not actions that are taken in any sense, they are placeholders for "no action" in the state representation so that the feature size is consistent, so that you can use a neural network. Another term might be "padding". Implement them as part of the starting state representation.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-28T08:01:51.257Z
Answers.Text: Can you share minimal working example code @Ykwk
Answers.UserDisplayName: Muhammad Yasirroni
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-02T13:35:25.827Z
Answers.LastActivityDate: 2021-09-02T13:35:25.827Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am planning to use OpenAI gym for my experiment in real life.
  In my experiment design, by the limits of a real-life scenario, I can only receive the state information or the rewards about 2-3 timesteps behind when the action has happened (in OpenAI gym term, ~3 cycles of <code>step(action)</code> function has occurred). For example, by the time the state at timestep <em>i</em> is observed, an action at timestep <em>i+3</em> would have happened.</p>
  <p>From how I perceive the function, <code>step(action)</code>, is that it needs to return <code>next_state, reward, done</code> every step. And the agent will learn from state -&gt; action -&gt; next state -&gt; reward tuple. So I was wondering if can I cache the action for future use along with the state with the correct time step in OpenAI gym? or delay the state observation/reward instead? Could the OpenAI be able to learn?</p>
  <p>I am experimenting with PPO TD3 SAC which all uses actor-critic networks. Would the network eventually be trained well enough to the point where it would still perform well with the delayed state observation?</p>
  
Tags: <reinforcement-learning><open-ai><gym>
Title: Delayed state observation or caching action in OpenAI gym. Can it still learn?
UserDisplayName: Ykwk
ThreadText: # Delayed state observation or caching action in OpenAI gym. Can it still learn?
  
  (Asked by: username_0 on 2021-09-01)
  
  I am planning to use OpenAI gym for my experiment in real life.
  In my experiment design, by the limits of a real-life scenario, I can only receive the state information or the rewards about 2-3 timesteps behind when the action has happened (in OpenAI gym term, ~3 cycles of `step(action)` function has occurred). For example, by the time the state at timestep *i* is observed, an action at timestep *i+3* would have happened.
  
  From how I perceive the function, `step(action)`, is that it needs to return `next_state, reward, done` every step. And the agent will learn from state -> action -> next state -> reward tuple. So I was wondering if can I cache the action for future use along with the state with the correct time step in OpenAI gym? or delay the state observation/reward instead? Could the OpenAI be able to learn?
  
  I am experimenting with PPO TD3 SAC which all uses actor-critic networks. Would the network eventually be trained well enough to the point where it would still perform well with the delayed state observation?
  
  **Tags:** reinforcement-learning, open-ai, gym
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-02. Score: 1 | Accepted answer
  
  To turn this delayed action approach into a normal, and theoretically valid MDP, add a "pending action resolution" array to the state representation, and ensure that the state transition (or `step` code) manages this array with each new action pushed into it, and the array shifting down as it goes. The array length should be minimum of the number of timesteps required to fully resolve a previous action.
  
  This allows your agent to see the actions it has recently performed and account for them in calculations without any adjustment needed to the core MDP formulation or standard agent designs.
  
  If delays are stochastic or vary (e.g. a scenario where a warehouse is ordering stock), then add more data to this pending action resolutions array as necessary, such as expected resolution time, or time since action was taken.
  
  > **Comments:**
  > **username_0**: From what I understand in implementation of your 'pending action resolution' array is:  First-In the step(action), take an action in real life. Second- Buffer/queue the action into the array with length of the delay step. Third - repeat fist&second until the action resolution array is completely filled. Last - With the full array, measure the state + reward associated with the oldest/last index of the pending action resolution array. Is this correct? If so, what do I return at the end of the step() function in the Second if I do not have the next state observed and the reward?  
  > **username_1**: @username_0: That's overcomplicated. The array of recent actions becomes just a normal part of the state. Initialise it with "null" actions so it has a fixed length. When the agent takes an action, push the id of that action onto the array, then shift the array to remove one old action (so the array stays the same length, and actions are stored in age order). Run all the rest of MDP and time step handling as normal for the agent - i.e. process value estimates, use replay memory for training etc on every time step.  
  > **username_0**: How can the 'initialization null' array of action be implemented, when every iteration of step() requires to return -next state -reward? Passing in 'null' action does not give me any return values for step() function. I don't think I can get the next action, without having the agent to go to next state reward.  
  > **username_1**: @username_0 These are not actions that are taken in any sense, they are placeholders for "no action" in the state representation so that the feature size is consistent, so that you can use a neural network. Another term might be "padding". Implement them as part of the starting state representation.  
  > **username_2**: Can you share minimal working example code @username_0  
  
  ---
  
  