Answers.Body: <p>Invalid actions are invalid because they cannot be taken <em>at all</em> by the agent. In board game scenarios that is usually because they are not allowed by the rules of the game, so a rules engine for that game can be used to filter actions down to the allowed set.</p>
  <p>An action that <em>can</em> be taken by an agent, but that would result in an unwanted reward is not considered <em>invalid</em>, and would therefore not be filtered by any approach that is called &quot;invalid action masking&quot;.</p>
  <p>Generally you would not want to remove an action from consideration entirely because something predicted a negative reward. That is already the role of the agent, to learn an optimal policy, and filtering out actions based on incomplete information will reduce the agent's ability to explore the space and find an optimal path.</p>
  <p>The example extension of CartPole does not define any invalid actions. With no other information to go on, all three actions are always valid. The agent can take them, and the environment will move into a new state. Contrast this to chess, where an action is to pick up an opponent's piece and move it as if it was some different piece. This is invalid because it breaks the rules of the game and completing such a move would stop the environment being a normal game of chess. Such moves are usually filtered - although another option is to allow them to be <em>requested</em>, give a negative reward leaving the game in the same state, and let the agent re-request its move (or, more drastically, have the agent immediately forfeit the game). Most game-playing agents filter and don't process invalid moves in any way, since learning the rules of the game is not the main objective.</p>
  <p>If you know more than the initial agent, it is <em>possible</em> to apply filtering to action choices any way you want to. The effect will be that the agent never experiences taking those actions, and never learns about them. So the same filtering must be left in place when you finish the learning, and should probably be considered a non-learning part of the agent.</p>
  <p>You can filter an action in this way using any logic you like. And you might decide to do so for instance, if outside of the reinforcement learning agent, you were able to predict something disastrous, such as a robot falling off a table causing damage.</p>
  <p>You can also &quot;truncate&quot; learning by having an exploration approach that cuts off for the worst action value predictions. This is not a usual approach, and has the same caveat as any other arbitrary action filtering - it may reduce the effectiveness of the agent, because maybe a trajectory that seemed bad initially would turn out to have a better end result once different actions are learned for later states. If you stop an agent from exploring that, it may never learn otherwise (although some off-policy and background planning approaches <em>might</em> have already collected enough data to self-correct).</p>
  <p>Truncating learning in that way would be unusual. But &quot;soft&quot; versions of that approach where the worst predictions get the least exploration, are more common. <a href="https://www.engati.com/glossary/thompson-sampling" rel="nofollow noreferrer">Thompson sampling</a> for reinforcement learning is one such approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-22T17:07:21.057Z
Answers.LastActivityDate: 2024-07-22T18:39:12.253Z
Answers.LastEditDate: 2024-07-22T18:39:12.253Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm reading about <em>Invalid Action Masking</em> in <em>RL</em> in order to use it in my PPO algorithm for a specific task.
  The problem is that I read such explanations: <a href="https://costa.sh/blog-a-closer-look-at-invalid-action-masking-in-policy-gradient-algorithms" rel="nofollow noreferrer">here</a>, <a href="https://boring-guy.sh/posts/masking-rl/" rel="nofollow noreferrer">here</a> and <a href="https://www.reddit.com/r/reinforcementlearning/comments/17l5b47/invalid_action_masking_when_action_space_is/" rel="nofollow noreferrer">here</a> the there the invalid action is choosen based on a <em>state</em> of the environment. My question is: is it possible to determine the invalidity of an action based on the reward?</p>
  <p>I want to make it clear based on a simple example. Let's take the old good Cartpole environment from OpenAI:</p>
  <p><a href="https://i.sstatic.net/K2Qr6JGy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/K2Qr6JGy.png" alt="enter image description here" /></a></p>
  <p>And let increase the <em>action space</em> from two to three. So there basically three discrete actions: <span class="math-container">$a_{t} \in [0, 1, 2]$</span>, where 0 means <em>left</em>, 1 means <em>right</em>, 2 means <em>do nothing</em>.</p>
  <p>How do I determine, which action - based on the state - should be masked out?</p>
  <p><strong>####   UPDATE   ####</strong></p>
  <p>Thanks to @Neil-Slater for answering the question. Reading his answer I understood, that my question was missleading and I didn't get to the point.
  The example I provided above is too simplified and didn't reflect exactly my question. So I try again this time using the picture I posted already in this question <a href="https://ai.stackexchange.com/questions/46276/what-is-the-best-strategy-to-train-a-model-with-multi-subgoals-in-the-same-env">here</a> few days ago. In the picture a crane moves from the <em>start</em> position to the <em>waypoint</em> <span class="math-container">$a$</span> as a first move. Now... the agent could theoretically go directly to <em>waypoint</em> <span class="math-container">$d$</span> ignoring all the waypoints in between. So I thought to <em>force</em> the agent to move vertically at first by invaliding the actions along the <span class="math-container">$x, y$</span> axes.</p>
  <p>Now... reading the answer from @Neil-Slater I can try to answer my question myself:</p>
  <ol>
  <li>The movement from <em>start</em> to <span class="math-container">$a$</span> does not require any invalidation of actions, because moving along the other axis is still <em>permitted</em> but not <em>wanted</em>. So the actions should not be masked at all. Reward should be enough.</li>
  <li>The movement form <em>start</em> to <span class="math-container">$a$</span> needs to be <em>forced</em> by using some masking of some sort, otherwise the agent would not learn a suitable policy for the vertical movement.</li>
  </ol>
  <p>I hope, I could now express myself better and clearer</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-22T16:09:45.877Z
Comments.Text: Why do you suppose that any of the actions be masked out in the cartpole environment? None of them seem "invalid" - the agent can clearly do all three, in all states. There is an important difference between "invalid" and "not good". Invalid actions are ones that you know that the agent cannot take, due to e.g. rules of a game, or constraints in the environment. Whilst going left when the pole is already fallen a lot to the right and was still falling would be *bad*, but it is not invalid
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-22T19:17:06.730Z
Comments.Text: Thanks to your comment and reply. I updated the answer above. Your answer is very dense (I'm still studying it), thus it  would be nide, if you could answer the question above with just  a solution 1 or solution 2, since I don't want you to reelaborate your very nice answer
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-22T19:31:07.947Z
Comments.Text: Having read the other question, I don't understand the environment setup. I get the feeling that you are overthinking the problem, but I cannot identify exactly what is wrong. Sorry
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-22T19:39:13.523Z
Comments.Text: I wanted to link to the picture and not to question in the other thread. I should have posted twice the same picture
Comments.UserDisplayName: Dave
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization><action-spaces>
Title: Is the Invalid Action Masking in RL related only to the state or can be dependent on the reward as well?
UserDisplayName: Dave
ThreadText: # Is the Invalid Action Masking in RL related only to the state or can be dependent on the reward as well?
  
  (Asked by: username_0 on 2024-07-22)
  
  I'm reading about *Invalid Action Masking* in *RL* in order to use it in my PPO algorithm for a specific task.
  The problem is that I read such explanations: [here](https://costa.sh/blog-a-closer-look-at-invalid-action-masking-in-policy-gradient-algorithms), [here](https://boring-guy.sh/posts/masking-rl/) and [here](https://www.reddit.com/r/reinforcementlearning/comments/17l5b47/invalid_action_masking_when_action_space_is/) the there the invalid action is choosen based on a *state* of the environment. My question is: is it possible to determine the invalidity of an action based on the reward?
  
  I want to make it clear based on a simple example. Let's take the old good Cartpole environment from OpenAI:
  
  [![enter image description here](https://i.sstatic.net/K2Qr6JGy.png)](https://i.sstatic.net/K2Qr6JGy.png)
  
  And let increase the *action space* from two to three. So there basically three discrete actions: $a\_{t} \in [0, 1, 2]$, where 0 means *left*, 1 means *right*, 2 means *do nothing*.
  
  How do I determine, which action - based on the state - should be masked out?
  
  **#### UPDATE ####**
  
  Thanks to @username_1-username_1 for answering the question. Reading his answer I understood, that my question was missleading and I didn't get to the point.
  The example I provided above is too simplified and didn't reflect exactly my question. So I try again this time using the picture I posted already in this question [here](https://ai.stackexchange.com/questions/46276/what-is-the-best-strategy-to-train-a-model-with-multi-subgoals-in-the-same-env) few days ago. In the picture a crane moves from the *start* position to the *waypoint* $a$ as a first move. Now... the agent could theoretically go directly to *waypoint* $d$ ignoring all the waypoints in between. So I thought to *force* the agent to move vertically at first by invaliding the actions along the $x, y$ axes.
  
  Now... reading the answer from @username_1-username_1 I can try to answer my question myself:
  
  1. The movement from *start* to $a$ does not require any invalidation of actions, because moving along the other axis is still *permitted* but not *wanted*. So the actions should not be masked at all. Reward should be enough.
  2. The movement form *start* to $a$ needs to be *forced* by using some masking of some sort, otherwise the agent would not learn a suitable policy for the vertical movement.
  
  I hope, I could now express myself better and clearer
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization, action-spaces
  
  > **Comments:**
  > **username_1**: Why do you suppose that any of the actions be masked out in the cartpole environment? None of them seem "invalid" - the agent can clearly do all three, in all states. There is an important difference between "invalid" and "not good". Invalid actions are ones that you know that the agent cannot take, due to e.g. rules of a game, or constraints in the environment. Whilst going left when the pole is already fallen a lot to the right and was still falling would be *bad*, but it is not invalid  
  > **username_0**: Thanks to your comment and reply. I updated the answer above. Your answer is very dense (I'm still studying it), thus it  would be nide, if you could answer the question above with just  a solution 1 or solution 2, since I don't want you to reelaborate your very nice answer  
  > **username_1**: Having read the other question, I don't understand the environment setup. I get the feeling that you are overthinking the problem, but I cannot identify exactly what is wrong. Sorry  
  > **username_0**: I wanted to link to the picture and not to question in the other thread. I should have posted twice the same picture  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-22. Score: 1
  
  Invalid actions are invalid because they cannot be taken *at all* by the agent. In board game scenarios that is usually because they are not allowed by the rules of the game, so a rules engine for that game can be used to filter actions down to the allowed set.
  
  An action that *can* be taken by an agent, but that would result in an unwanted reward is not considered *invalid*, and would therefore not be filtered by any approach that is called "invalid action masking".
  
  Generally you would not want to remove an action from consideration entirely because something predicted a negative reward. That is already the role of the agent, to learn an optimal policy, and filtering out actions based on incomplete information will reduce the agent's ability to explore the space and find an optimal path.
  
  The example extension of CartPole does not define any invalid actions. With no other information to go on, all three actions are always valid. The agent can take them, and the environment will move into a new state. Contrast this to chess, where an action is to pick up an opponent's piece and move it as if it was some different piece. This is invalid because it breaks the rules of the game and completing such a move would stop the environment being a normal game of chess. Such moves are usually filtered - although another option is to allow them to be *requested*, give a negative reward leaving the game in the same state, and let the agent re-request its move (or, more drastically, have the agent immediately forfeit the game). Most game-playing agents filter and don't process invalid moves in any way, since learning the rules of the game is not the main objective.
  
  If you know more than the initial agent, it is *possible* to apply filtering to action choices any way you want to. The effect will be that the agent never experiences taking those actions, and never learns about them. So the same filtering must be left in place when you finish the learning, and should probably be considered a non-learning part of the agent.
  
  You can filter an action in this way using any logic you like. And you might decide to do so for instance, if outside of the reinforcement learning agent, you were able to predict something disastrous, such as a robot falling off a table causing damage.
  
  You can also "truncate" learning by having an exploration approach that cuts off for the worst action value predictions. This is not a usual approach, and has the same caveat as any other arbitrary action filtering - it may reduce the effectiveness of the agent, because maybe a trajectory that seemed bad initially would turn out to have a better end result once different actions are learned for later states. If you stop an agent from exploring that, it may never learn otherwise (although some off-policy and background planning approaches *might* have already collected enough data to self-correct).
  
  Truncating learning in that way would be unusual. But "soft" versions of that approach where the worst predictions get the least exploration, are more common. [Thompson sampling](https://www.engati.com/glossary/thompson-sampling) for reinforcement learning is one such approach.
  
  
  ---
  
  