Answers.Body: <p>This is actually an implementation choice, and will depend on how you chose to represent the agent's model of the function that maps from states to actions.</p>
  <p>If you explicitly represent the entire state space, as you might chose to do with simple benchmark problems that you solve by directly solving an MDP with something like <a href="https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node19.html" rel="nofollow noreferrer">value iteration</a>, then you can also easily explicitly represent exactly the set of actions that the agent can perform in each state, and the agent can learn the expected value of just taking those actions.</p>
  <p>If your state space is very large, you may not be able to represent it explicitly, and your agent is more likely to use some approximation of the value function or its policy, as is commonly done in <a href="https://en.wikipedia.org/wiki/Q-learning" rel="nofollow noreferrer">Q-Learning</a>. Here, it is often preferable to define your model of the environment so that taking an invalid action in a state causes some well-defined outcome, or causes the agent to randomly re-select its actions until it ends up picking a valid one. The agent will eventually learn that selecting an invalid action leads to bad outcomes, without &quot;realizing&quot; that the action is invalid.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-25T02:40:20.897Z
Answers.LastActivityDate: 2020-11-17T21:54:09.500Z
Answers.LastEditDate: 2020-11-17T21:54:09.500Z
Answers.UserDisplayName: John Doucette
Body: <p>I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for each state, or is the agent only aware of the entire action space (in which case I guess the environment needs to discard invalid actions)?</p>
  <p>I presume the answer is yes, but I would like to confirm.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-17T21:55:49.053Z
Comments.Text: [Here](https://ai.stackexchange.com/q/7755/2444) is a related question.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><markov-decision-process><action-spaces>
Title: Is the agent aware of a possible different set of actions for each state?
UserDisplayName: Francis Chang
ThreadText: # Is the agent aware of a possible different set of actions for each state?
  
  (Asked by: username_0 on 2019-09-24)
  
  I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for each state, or is the agent only aware of the entire action space (in which case I guess the environment needs to discard invalid actions)?
  
  I presume the answer is yes, but I would like to confirm.
  
  **Tags:** reinforcement-learning, markov-decision-process, action-spaces
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/7755/2444) is a related question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-25. Score: 1
  
  This is actually an implementation choice, and will depend on how you chose to represent the agent's model of the function that maps from states to actions.
  
  If you explicitly represent the entire state space, as you might chose to do with simple benchmark problems that you solve by directly solving an MDP with something like [value iteration](https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node19.html), then you can also easily explicitly represent exactly the set of actions that the agent can perform in each state, and the agent can learn the expected value of just taking those actions.
  
  If your state space is very large, you may not be able to represent it explicitly, and your agent is more likely to use some approximation of the value function or its policy, as is commonly done in [Q-Learning](https://en.wikipedia.org/wiki/Q-learning). Here, it is often preferable to define your model of the environment so that taking an invalid action in a state causes some well-defined outcome, or causes the agent to randomly re-select its actions until it ends up picking a valid one. The agent will eventually learn that selecting an invalid action leads to bad outcomes, without "realizing" that the action is invalid.
  
  
  ---
  
  