Answers.Body: <p>In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.</p>
  <p>It may help if you think through what the actual consequences for making the &quot;wrong&quot; move are in your environment. There are a few self-consistent options:</p>
  <ul>
  <li><p>The move simply cannot be made and count as playing the game as intended. In which case, do not allow the agent to make that choice. You can achieve that by filtering the list of choices that the agent is allowed to make. In DQN that will mean supplying an action mask to the agent based on the state so it does not include the action at the stage it makes a choice. This &quot;available actions&quot; function is usually coded as part of the environment.</p>
  </li>
  <li><p>The move can be attempted, but results in no change to state (e.g. the agent bumps into a wall). If the goal is to reach a certain state in shortest possible time, then you will typically have 0 reward and a discount factor, or negative reward for each attempted action. Either way, the agent should learn that the move was wasted and avoid it after a few iterations.</p>
  </li>
  <li><p>The move can be attempted, but results in disaster (e.g. the agent falls off a cliff). This is the case where a large negative reward plus ending the episode is appropriate. However, don't use infinite rewards, positive or negative, because that will cause significant problems with numeric stability. Simply large enough to offset any interim positive rewards associated with that direction should be adequate. For a simple goal-seeking environment with no other positive rewards than reaching the goal, ending the episode early is already enough.</p>
  </li>
  </ul>
  <p>When you don't have a toy environment where you get to decide, then the three basic scenarios above can still help. For instance, in most board games we are not interested in having the agent learn the rules for valid moves when they are already supplied by the environment, so the first scenario applies - only select actions from the valid ones as provided by the environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-20T17:23:14.047Z
Answers.Text: Thanks! In my toy project, the first solution is appropriate. What should I do if I want to add a mask to prevent the agent to behave "downward" and "leftward"?  My current solution is to end this episode immediately if the agent makes those actions and gives a fairly large negative reward. I think an action mask is better than my current solution. How can I implement the action mask in the environment? My DQN network inputs states and outputs Q(s, a).
Answers.UserDisplayName: o_yeah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-20T17:25:49.273Z
Answers.Text: Will it be similar to adding a manual drop-out, and keeps the out-put "downward" and "leftward" neurons always inactive when in this beginning state?
Answers.UserDisplayName: o_yeah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-20T17:56:37.450Z
Answers.Text: @o_yeah: It depends on how you have implemented the neural network. But either way, the filtering is done outside of the neural network, by the learning algorithm when it selects an action. If your NN takes s,a as input in a mini-batch to find the best action, then filter the list of action choices before making the mini-batch. If your NN outputs an array of Q values for every action, write code to somehow ignore values of non-valid actions (e.g. set value to -inf before argmax). In both cases you will need a function defined in the environment that tells you what the valid action choices are.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-19T08:20:41.047Z
Answers.LastActivityDate: 2020-11-14T18:07:54.587Z
Answers.LastEditDate: 2020-11-14T18:07:54.587Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a <span class="math-container">$3 \times 3$</span> grid world.</p>
  <p>I plan to use DQN to do this. I'm having trouble handling the starting point: what if the Q network's prediction is telling the agent to move downward (or leftward) at the beginning?</p>
  <p>Shall I program the environment to immediately give a <span class="math-container">$-\infty$</span> reward and end this episode? Will this penalty make the agent &quot;fear&quot; of moving left again in the future, even if moving left is a possible choice?</p>
  <p>Any suggestions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-14T18:06:23.460Z
Comments.Text: [Here](https://ai.stackexchange.com/q/2980/2444) is a similar question in the context of REINFORCE and [here](https://ai.stackexchange.com/q/3403/2444) is another similar question but in the context of a specific reward function (and DQN).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><dqn><reward-design><reward-functions>
Title: How should I handle invalid actions in a grid world?
UserDisplayName: o_yeah
ThreadText: # How should I handle invalid actions in a grid world?
  
  (Asked by: username_0 on 2020-05-19)
  
  I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world.
  
  I plan to use DQN to do this. I'm having trouble handling the starting point: what if the Q network's prediction is telling the agent to move downward (or leftward) at the beginning?
  
  Shall I program the environment to immediately give a $-\infty$ reward and end this episode? Will this penalty make the agent "fear" of moving left again in the future, even if moving left is a possible choice?
  
  Any suggestions?
  
  **Tags:** reinforcement-learning, q-learning, dqn, reward-design, reward-functions
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/2980/2444) is a similar question in the context of REINFORCE and [here](https://ai.stackexchange.com/q/3403/2444) is another similar question but in the context of a specific reward function (and DQN).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-19. Score: 3 | Accepted answer
  
  In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.
  
  It may help if you think through what the actual consequences for making the "wrong" move are in your environment. There are a few self-consistent options:
  
  * The move simply cannot be made and count as playing the game as intended. In which case, do not allow the agent to make that choice. You can achieve that by filtering the list of choices that the agent is allowed to make. In DQN that will mean supplying an action mask to the agent based on the state so it does not include the action at the stage it makes a choice. This "available actions" function is usually coded as part of the environment.
  * The move can be attempted, but results in no change to state (e.g. the agent bumps into a wall). If the goal is to reach a certain state in shortest possible time, then you will typically have 0 reward and a discount factor, or negative reward for each attempted action. Either way, the agent should learn that the move was wasted and avoid it after a few iterations.
  * The move can be attempted, but results in disaster (e.g. the agent falls off a cliff). This is the case where a large negative reward plus ending the episode is appropriate. However, don't use infinite rewards, positive or negative, because that will cause significant problems with numeric stability. Simply large enough to offset any interim positive rewards associated with that direction should be adequate. For a simple goal-seeking environment with no other positive rewards than reaching the goal, ending the episode early is already enough.
  
  When you don't have a toy environment where you get to decide, then the three basic scenarios above can still help. For instance, in most board games we are not interested in having the agent learn the rules for valid moves when they are already supplied by the environment, so the first scenario applies - only select actions from the valid ones as provided by the environment.
  
  > **Comments:**
  > **username_0**: Thanks! In my toy project, the first solution is appropriate. What should I do if I want to add a mask to prevent the agent to behave "downward" and "leftward"?  My current solution is to end this episode immediately if the agent makes those actions and gives a fairly large negative reward. I think an action mask is better than my current solution. How can I implement the action mask in the environment? My DQN network inputs states and outputs Q(s, a).  
  > **username_0**: Will it be similar to adding a manual drop-out, and keeps the out-put "downward" and "leftward" neurons always inactive when in this beginning state?  
  > **username_2**: @o_yeah: It depends on how you have implemented the neural network. But either way, the filtering is done outside of the neural network, by the learning algorithm when it selects an action. If your NN takes s,a as input in a mini-batch to find the best action, then filter the list of action choices before making the mini-batch. If your NN outputs an array of Q values for every action, write code to somehow ignore values of non-valid actions (e.g. set value to -inf before argmax). In both cases you will need a function defined in the environment that tells you what the valid action choices are.  
  
  ---
  
  