Answers.Body: <p>You could code your agent's policy to never select impossible actions.</p>
  <p>Your other question implies that you are writing your own behaviour policy function (e.g. you asked about implementing a softmax policy). The behaviour policy function must take the current state as input, and output an action choice.</p>
  <p>For your environment to work, <em>some</em> code somewhere presumably knows which actions are &quot;impossible&quot;.</p>
  <p>Rule the impossible actions out in the policy by calling the environment code that knows about them, and assigning probability zero to them, then you don't need to care that the rest of the framework is allocating places in the Q table for them.</p>
  <p>An alternative to querying the environment directly from inside the agent would be to pre-populate the Q table with large negative values for all impossible combinations, and set probability to zero in your policy for any Q value this low or lower. You should be able to choose a really large negative value for this purpose (perhaps even <code>-Inf</code> if supported) to avoid any interference from the rest of the environment's reward structure. The advantage of this approach is that an eventual <em>greedy</em> agent would not need to query the environment code, it could use the Q table as-is.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T13:30:16.307Z
Answers.LastActivityDate: 2022-05-19T13:57:09.783Z
Answers.LastEditDate: 2022-05-19T13:57:09.783Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am using Q-learning in julia language.</p>
  <p>Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I can't use a function that, given a state, returns all the possible actions. In order to solve it I am using a dummy state which is terminal and a bad reward.</p>
  <p>When the agent tries to take impossible actions, what  is the difference between using a dummy state which is terminal and remaining in the same state (+bad reward) until the end of the episode? Other possible solutions?</p>
  <p>Specifically, how can I either avoid defining an impossible action or alternatively define it as an impossible action?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-19T08:11:19.380Z
Comments.Text: What do you mean by "Because of the solver"? Do you mean "Because of Q-learning"? Also, which function are you referring to when you say "a function that, given a state, returns all the possible actions"? A policy? Also, are you using the a "dummy state and a bad reward" for what?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-19T08:13:28.413Z
Comments.Text: I am using Julia language and its q learning solver doesn’t work with state dependent actions. The function was made by me
Comments.UserDisplayName: Aquila
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-19T08:15:26.923Z
Comments.Text: By "state dependent actions", do you mean that your Julia implementation doesn't accept an action that depend on the state (whatever that means - maybe you mean that the function you're calling doesn't accept a state as input in order to product an action)? I don't understand your question at all. **Edit your post to try to describe your problem more in detail and clearly**. Right now, it seems that you have a problem related to impossible/illegal actions, but, apart from that, it's not fully clear what your problem is (at least to me).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-19T08:30:12.843Z
Comments.Text: Done, I have changed the description
Comments.UserDisplayName: Aquila
Tags: <reinforcement-learning><q-learning><state-spaces><action-spaces><state-representations>
Title: How to manage impossible actions?
UserDisplayName: Aquila
ThreadText: # How to manage impossible actions?
  
  (Asked by: username_0 on 2022-05-19)
  
  I am using Q-learning in julia language.
  
  Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I can't use a function that, given a state, returns all the possible actions. In order to solve it I am using a dummy state which is terminal and a bad reward.
  
  When the agent tries to take impossible actions, what is the difference between using a dummy state which is terminal and remaining in the same state (+bad reward) until the end of the episode? Other possible solutions?
  
  Specifically, how can I either avoid defining an impossible action or alternatively define it as an impossible action?
  
  **Tags:** reinforcement-learning, q-learning, state-spaces, action-spaces, state-representations
  
  > **Comments:**
  > **username_1**: What do you mean by "Because of the solver"? Do you mean "Because of Q-learning"? Also, which function are you referring to when you say "a function that, given a state, returns all the possible actions"? A policy? Also, are you using the a "dummy state and a bad reward" for what?  
  > **username_0**: I am using Julia language and its q learning solver doesn’t work with state dependent actions. The function was made by me  
  > **username_1**: By "state dependent actions", do you mean that your Julia implementation doesn't accept an action that depend on the state (whatever that means - maybe you mean that the function you're calling doesn't accept a state as input in order to product an action)? I don't understand your question at all. **Edit your post to try to describe your problem more in detail and clearly**. Right now, it seems that you have a problem related to impossible/illegal actions, but, apart from that, it's not fully clear what your problem is (at least to me).  
  > **username_0**: Done, I have changed the description  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-19. Score: 1
  
  You could code your agent's policy to never select impossible actions.
  
  Your other question implies that you are writing your own behaviour policy function (e.g. you asked about implementing a softmax policy). The behaviour policy function must take the current state as input, and output an action choice.
  
  For your environment to work, *some* code somewhere presumably knows which actions are "impossible".
  
  Rule the impossible actions out in the policy by calling the environment code that knows about them, and assigning probability zero to them, then you don't need to care that the rest of the framework is allocating places in the Q table for them.
  
  An alternative to querying the environment directly from inside the agent would be to pre-populate the Q table with large negative values for all impossible combinations, and set probability to zero in your policy for any Q value this low or lower. You should be able to choose a really large negative value for this purpose (perhaps even `-Inf` if supported) to avoid any interference from the rest of the environment's reward structure. The advantage of this approach is that an eventual *greedy* agent would not need to query the environment code, it could use the Q table as-is.
  
  
  ---
  
  