Answers.Body: <p>One way to handle an arbitrarily large sequence is by adding a STOP signal as one possible token in the sequence, just like LSTM.</p>
  <p>So you could divide your game in turns:</p>
  <ul>
  <li>What you now call a single action (composed by multiple sub-actions) would become a turn.</li>
  <li>Now, you can have as many actions you'd like inside a turn.</li>
  <li>Each action is simply a list accumulated inside the environment, but won't evaluate the game yet.</li>
  <li>When the player is satisfied with their actions, they can call the action: &quot;End Turn&quot;.</li>
  <li>When turn ends, you can concatenate every sub-actions, evaluate the game and proceed as usual.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-20T15:51:13.793Z
Answers.LastActivityDate: 2021-08-20T15:51:13.793Z
Answers.UserDisplayName: Andre Goulart
Body: <p>I'm attempting to design an action space in OpenAI's gym and hitting the following roadblock. I've looked at <a href="https://stackoverflow.com/q/45001361/3924118">this post</a> which is closely related but subtly different.</p>
  
  <p>The environment I'm writing needs to allow an agent to make between <span class="math-container">$1$</span> and <span class="math-container">$n$</span> sub-actions in each step. Leaving it up to the agent to decide how many sub-actions it wants to take. So, something like (sub-action-category, sub-action-id, action) where the agent can specify between <span class="math-container">$1$</span> and <span class="math-container">$n$</span> such tuples.</p>
  
  <p>It doesn't seem possible to define a <code>Box</code> space without specifying bounds on the shape which is what I need here. I'm trying to avoid defining an action space where each sub-action is explicitly enumerated by the environment like (action) tuple with n entries for each sub-action.</p>
  
  <p>Are there any other spaces I could use to dynamically scale the space?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-19T09:51:51.737Z
Comments.Text: Is the last element of the tuple `(sub-action-category, sub-action-id, action)` a discrete or continuous value? What is its dimension? `gym` offers the class `MultiDiscrete` for a similar use case. You could also write your own action type if you wanted to, but imo the straightest way to deal with this is to have actions to be instances of some suitably defined Python class you can document etc. and deal with the details of the execution to the `Environment` subclass.
Comments.UserDisplayName: Miquel Ramirez
Tags: <reinforcement-learning><ai-design><open-ai><gym>
Title: How to define an action space when an agent can take multiple sub-actions in a step?
UserDisplayName: Seyed Moein Ayyoubzadeh
ThreadText: # How to define an action space when an agent can take multiple sub-actions in a step?
  
  (Asked by: username_0 on 2020-02-19)
  
  I'm attempting to design an action space in OpenAI's gym and hitting the following roadblock. I've looked at [this post](https://stackoverflow.com/q/45001361/3924118) which is closely related but subtly different.
  
  The environment I'm writing needs to allow an agent to make between $1$ and $n$ sub-actions in each step. Leaving it up to the agent to decide how many sub-actions it wants to take. So, something like (sub-action-category, sub-action-id, action) where the agent can specify between $1$ and $n$ such tuples.
  
  It doesn't seem possible to define a `Box` space without specifying bounds on the shape which is what I need here. I'm trying to avoid defining an action space where each sub-action is explicitly enumerated by the environment like (action) tuple with n entries for each sub-action.
  
  Are there any other spaces I could use to dynamically scale the space?
  
  **Tags:** reinforcement-learning, ai-design, open-ai, gym
  > **Comments:**
  > **username_1**: Is the last element of the tuple `(sub-action-category, sub-action-id, action)` a discrete or continuous value? What is its dimension? `gym` offers the class `MultiDiscrete` for a similar use case. You could also write your own action type if you wanted to, but imo the straightest way to deal with this is to have actions to be instances of some suitably defined Python class you can document etc. and deal with the details of the execution to the `Environment` subclass.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-20. Score: 2 | Accepted answer
  
  One way to handle an arbitrarily large sequence is by adding a STOP signal as one possible token in the sequence, just like LSTM.
  
  So you could divide your game in turns:
  
  * What you now call a single action (composed by multiple sub-actions) would become a turn.
  * Now, you can have as many actions you'd like inside a turn.
  * Each action is simply a list accumulated inside the environment, but won't evaluate the game yet.
  * When the player is satisfied with their actions, they can call the action: "End Turn".
  * When turn ends, you can concatenate every sub-actions, evaluate the game and proceed as usual.
  
  
  ---
  
  