Answers.Body: <p>There are two relevant neural network designs for DQN:</p>
  <ul>
  <li><p>Model q function directly <span class="math-container">$Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$</span>, so neural network has concatenated input of state and action, and outputs a single real value. This is arguably the more natural fit to Q learning, but can be inefficient.</p>
  </li>
  <li><p>Model all q values for given state <span class="math-container">$Q(s,\cdot): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$</span>, so neural network takes input of current state and outputs all action values related to that state as a vector.</p>
  </li>
  </ul>
  <p>For the first architecture, you can decide which actions to evaluate by how you construct the minibatch. You pre-filter to the allowed actions for each state.</p>
  <p>For the second architecture, you must post-filter the action values to those allowed by the state.</p>
  <p>There are other possibilities for constructing variable-length inputs and outputs to neural networks - e.g. using RNNs. However, these are normally not worth the extra effort. A pre- or post- filter on the actions for a NN that can process the whole action space (including impossible actions) is all you usually need. Don't worry that the neural network may calculate some non-needed or nonsense values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T18:35:07.027Z
Answers.Text: I am using the second architecture, can you elaborate more on how to post-filter the action values?
Answers.UserDisplayName: ycenycute
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T18:44:50.513Z
Answers.Text: @TracyYang: That will depend on your software library. The point is to ensure that they are not considered when choosing an action to take (or looking at the maximum action value from a state for updates). So for example, if you are using a max or argmax function to select, then set the unwanted values to `-inf` before running it.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T20:42:25.967Z
Answers.Text: So, the action size is still 5, but when it should be 0,3,4 only, then set values corresponding to 1 and 2 to `-inf` first before using argmax. Is this correct?
Answers.UserDisplayName: ycenycute
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-11T07:23:09.197Z
Answers.Text: @TracyYang Yes that is correct
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-09T20:25:25.910Z
Answers.LastActivityDate: 2021-01-09T20:33:40.707Z
Answers.LastEditDate: 2021-01-09T20:33:40.707Z
Answers.UserDisplayName: Neil Slater
Body: <p>I would like to employ DQN to solve a constrained MDP problem. The problem has constraints on action space. At different time steps till the end, the available actions are different. It has different possibilities as below.</p>
  <ul>
  <li>0, 1, 2, 3, 4</li>
  <li>0, 2, 3, 4</li>
  <li>0, 3, 4</li>
  <li>0, 4</li>
  </ul>
  <p>Does this mean I need to learn 4 different Q networks for these possibilities? Also, correct me if I am wrong, it looks like if I specify the action size is 3, then it automatically assumes the actions are <code>0, 1, 2</code>, but, in my case, it should be <code>0, 3, 4</code>. How shall I implement this?</p>
  
Tags: <reinforcement-learning><dqn><action-spaces><constrained-optimization>
Title: How to use DQN when the action space can be different at different time steps?
UserDisplayName: ycenycute
ThreadText: # How to use DQN when the action space can be different at different time steps?
  
  (Asked by: username_0 on 2021-01-09)
  
  I would like to employ DQN to solve a constrained MDP problem. The problem has constraints on action space. At different time steps till the end, the available actions are different. It has different possibilities as below.
  
  * 0, 1, 2, 3, 4
  * 0, 2, 3, 4
  * 0, 3, 4
  * 0, 4
  
  Does this mean I need to learn 4 different Q networks for these possibilities? Also, correct me if I am wrong, it looks like if I specify the action size is 3, then it automatically assumes the actions are `0, 1, 2`, but, in my case, it should be `0, 3, 4`. How shall I implement this?
  
  **Tags:** reinforcement-learning, dqn, action-spaces, constrained-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-09. Score: 2 | Accepted answer
  
  There are two relevant neural network designs for DQN:
  
  * Model q function directly $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, so neural network has concatenated input of state and action, and outputs a single real value. This is arguably the more natural fit to Q learning, but can be inefficient.
  * Model all q values for given state $Q(s,\cdot): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$, so neural network takes input of current state and outputs all action values related to that state as a vector.
  
  For the first architecture, you can decide which actions to evaluate by how you construct the minibatch. You pre-filter to the allowed actions for each state.
  
  For the second architecture, you must post-filter the action values to those allowed by the state.
  
  There are other possibilities for constructing variable-length inputs and outputs to neural networks - e.g. using RNNs. However, these are normally not worth the extra effort. A pre- or post- filter on the actions for a NN that can process the whole action space (including impossible actions) is all you usually need. Don't worry that the neural network may calculate some non-needed or nonsense values.
  
  > **Comments:**
  > **username_0**: I am using the second architecture, can you elaborate more on how to post-filter the action values?  
  > **username_1**: @TracyYang: That will depend on your software library. The point is to ensure that they are not considered when choosing an action to take (or looking at the maximum action value from a state for updates). So for example, if you are using a max or argmax function to select, then set the unwanted values to `-inf` before running it.  
  > **username_0**: So, the action size is still 5, but when it should be 0,3,4 only, then set values corresponding to 1 and 2 to `-inf` first before using argmax. Is this correct?  
  > **username_1**: @TracyYang Yes that is correct  
  
  ---
  
  