Answers.Body: <p>You can definitely define an environment that accepts more types of action, including actions that take multiple steps in a direction.</p>
  
  <p>The first thing you would need to do is implement support for that action in the environment. That is not really a reinforcement learning issue, but like implementing the rules of a board game. You will need to decide things such as what happens if the move would be blocked - does the move succeed up the point of being blocked, does it fail completely, is the reward lower depending on how much the agent tries to overshoot, etc.</p>
  
  <p>After you do that, you will want to write an agent that can choose the new actions. You have a few choices here:</p>
  
  <ul>
  <li><p>Simplest would be to enumerate all the choices separately and continue to use the same kind of agent as you already have. So instead of <span class="math-container">$\{U, D, L, R\}$</span> you might have <span class="math-container">$\{U1, U2, U3, D1, D2, D3, L1, L2, L3, R1, R2, R3\}$</span>.</p></li>
  <li><p>If you want to take advantage of generalisation between similar actions (e.g. that action <span class="math-container">$U3$</span> is similar to <span class="math-container">$U2$</span> and also to <span class="math-container">$R3$</span>), then you can use some form of coding for the action, such as the relative x,y movement that it is attempting. So you could express <span class="math-container">$U2$</span> as <span class="math-container">$(0,2)$</span> and <span class="math-container">$L3$</span> as <span class="math-container">$(-3,0)$</span>. For that to then work with Q values, you cannot easily use a table. Instead, you would need to use function approximation, for example a neural network, so you can implement <span class="math-container">$q(s,a)$</span> as a parametric function - combining <span class="math-container">$s,a$</span> into the input vector, and learn the parameters to that the neural network outputs the correct action value. This is what the Q learning variation DQN can do, as well as other similar RL algorithms that use neural networks.</p></li>
  </ul>
  
  <p>Using a neural network, instead of tabular Q-learning, is not something you see often with grid world environments. It is a step up in complexity, but it is often required if state space or action space becomes large and might benefit from the generalisation possible from trainable function approximators.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T21:19:40.627Z
Answers.Text: @ NeilSlater Ah yes I expected that I would have to do something like the first bullet point. Yes, I saw in different videos and tutorials that tabular Q-learning is better when dealing with GridWorld environments. So would it not be possible for an environment itself to decide how many steps it should take after choosing a direction to take? Additionally, could you please explain the second bullet point in a bit more detail, specifically the first sentence of it? Thank You!
Answers.UserDisplayName: Huzaifah Shamim
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T21:25:38.533Z
Answers.Text: @HuzaifahShamim I added a couple of examples to the second bullet point
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T22:12:11.773Z
Answers.Text: @ NeilSlater, how is expressing U2 and L3 that way different then if I was to do it via QTable? I was thinking that that is how I would express it if I was to do it like the first bullet point.
Answers.UserDisplayName: Huzaifah Shamim
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T22:41:59.690Z
Answers.Text: @HuzaifahShamim Well you can always enumerate and put into a table, and/or one-hot encode if you wish. But the numerical coordinates version can *also* be used as input into an approximation algorithm, supporting generalisation - something you generally want when the combined state and action space becomes large.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T22:44:54.933Z
Answers.Text: There is no requirement to try second bullet point, if you don't feel you need it. As explained just increasing the enumerations is simplest and just fine up to the point when you have too many options to handle. 12 or 16 or 20 action choices is not many. But 2000 might be too much if the agent could choose to move anything from 1 to 500 steps in any direction in a single time step
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-18T19:55:52.490Z
Answers.Text: " But the numerical coordinates version can also be used as input into an approximation algorithm, supporting generalisation"
  
  Could you please explain this part and/or refer me to a video so I can better udnerstand it?
Answers.UserDisplayName: Huzaifah Shamim
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-18T20:46:07.070Z
Answers.Text: @HuzaifahShamim: The main point of using approximations like neural networks is the suport of generalisation. After you train a neural network on a set of data, it is possible to give it some new data that it has not seen before and it can still produce an output - a well-trained NN will output an accurate value for whatever you have trained it for. A table of average values for inputs you have seen before - equivalent to tabular Q learning - cannot possibly do that.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T20:43:45.983Z
Answers.LastActivityDate: 2020-06-16T21:24:37.297Z
Answers.LastEditDate: 2020-06-16T21:24:37.297Z
Answers.UserDisplayName: Neil Slater
Body: <p>In a classic <strong>GridWorld Environment</strong> where the possible <strong>actions</strong> of an agent are (Up, Down, Left, Right), can another potential output of Action be "x amount of steps" where the agent takes 2,3,.. steps in the direction (U,D,L,R) that it chooses? If so, how would one go about doing it?</p>
  
Tags: <reinforcement-learning>
Title: Additional (Potential) Action for Agent in MazeGrid Environment (Reinforcement Learning)
UserDisplayName: Huzaifah Shamim
ThreadText: # Additional (Potential) Action for Agent in MazeGrid Environment (Reinforcement Learning)
  
  (Asked by: username_0 on 2020-06-16)
  
  In a classic **GridWorld Environment** where the possible **actions** of an agent are (Up, Down, Left, Right), can another potential output of Action be "x amount of steps" where the agent takes 2,3,.. steps in the direction (U,D,L,R) that it chooses? If so, how would one go about doing it?
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-16. Score: 0 | Accepted answer
  
  You can definitely define an environment that accepts more types of action, including actions that take multiple steps in a direction.
  
  The first thing you would need to do is implement support for that action in the environment. That is not really a reinforcement learning issue, but like implementing the rules of a board game. You will need to decide things such as what happens if the move would be blocked - does the move succeed up the point of being blocked, does it fail completely, is the reward lower depending on how much the agent tries to overshoot, etc.
  
  After you do that, you will want to write an agent that can choose the new actions. You have a few choices here:
  
  * Simplest would be to enumerate all the choices separately and continue to use the same kind of agent as you already have. So instead of $\{U, D, L, R\}$ you might have $\{U1, U2, U3, D1, D2, D3, L1, L2, L3, R1, R2, R3\}$.
  * If you want to take advantage of generalisation between similar actions (e.g. that action $U3$ is similar to $U2$ and also to $R3$), then you can use some form of coding for the action, such as the relative x,y movement that it is attempting. So you could express $U2$ as $(0,2)$ and $L3$ as $(-3,0)$. For that to then work with Q values, you cannot easily use a table. Instead, you would need to use function approximation, for example a neural network, so you can implement $q(s,a)$ as a parametric function - combining $s,a$ into the input vector, and learn the parameters to that the neural network outputs the correct action value. This is what the Q learning variation DQN can do, as well as other similar RL algorithms that use neural networks.
  
  Using a neural network, instead of tabular Q-learning, is not something you see often with grid world environments. It is a step up in complexity, but it is often required if state space or action space becomes large and might benefit from the generalisation possible from trainable function approximators.
  
  > **Comments:**
  > **username_0**: @ username_1 Ah yes I expected that I would have to do something like the first bullet point. Yes, I saw in different videos and tutorials that tabular Q-learning is better when dealing with GridWorld environments. So would it not be possible for an environment itself to decide how many steps it should take after choosing a direction to take? Additionally, could you please explain the second bullet point in a bit more detail, specifically the first sentence of it? Thank You!  
  > **username_1**: @username_0 I added a couple of examples to the second bullet point  
  > **username_0**: @ username_1, how is expressing U2 and L3 that way different then if I was to do it via QTable? I was thinking that that is how I would express it if I was to do it like the first bullet point.  
  > **username_1**: @username_0 Well you can always enumerate and put into a table, and/or one-hot encode if you wish. But the numerical coordinates version can *also* be used as input into an approximation algorithm, supporting generalisation - something you generally want when the combined state and action space becomes large.  
  > **username_1**: There is no requirement to try second bullet point, if you don't feel you need it. As explained just increasing the enumerations is simplest and just fine up to the point when you have too many options to handle. 12 or 16 or 20 action choices is not many. But 2000 might be too much if the agent could choose to move anything from 1 to 500 steps in any direction in a single time step  
  > **username_0**: " But the numerical coordinates version can also be used as input into an approximation algorithm, supporting generalisation"  Could you please explain this part and/or refer me to a video so I can better udnerstand it?  
  > **username_1**: @username_0: The main point of using approximations like neural networks is the suport of generalisation. After you train a neural network on a set of data, it is possible to give it some new data that it has not seen before and it can still produce an output - a well-trained NN will output an accurate value for whatever you have trained it for. A table of average values for inputs you have seen before - equivalent to tabular Q learning - cannot possibly do that.  
  
  ---
  
  