Body: <p>I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
  However, there is a constraint that one task should be served by only one server.How should I design the action space?</p>
  <p>So when I design the decision-making action space,I don't know which design is better:</p>
  <pre><code>        self.action_space = spaces.MultiDiscrete([2]*(M*N)) #action space: [a_11,a_12,...,a_1N,a_M1,a_M2,...,a_MN]  M*N discrete vector,a_mn in [0,1]
          self.action_space = spaces.MultiDiscrete([M]*N) #action space: [a_1,a_2,...,a_N] N discrete vector, a_n in [0,M-1]
  </code></pre>
  <p>In first way, the variable a_mn stands for if the n-th task  is offloaded to m-th server or not. While in the second way,a_n represents which server should be chosen by the n-th task.</p>
  <p>If the first way is better, how can I implement the constraints mentioned before, since there is lots of infeasible actions in the action space?</p>
  <p>And I was confused in both situation if it is possible to use the DQN to produce the Q value of all action from this action space?
  Thank a lot</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T18:01:24.140Z
Comments.Text: maybe https://towardsdatascience.com/action-masking-with-rllib-5e4bec5e7505?
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-02T08:05:18.487Z
Comments.Text: It may be easier to allow infeasible actions to exist in your problem. Instead, set them to transition to a terminal state returning a highly negative reward. This should prevent such actions from being selected at evaluation, however, it depends on whether it would be ok to select these sort of actions in training.
Comments.UserDisplayName: quest ions
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-05T14:27:01.553Z
Comments.Text: Thanks a lot. Do you mean the first way is better, even there is lots of actions infeasible in the first action space?
Comments.UserDisplayName: Reese
Tags: <reinforcement-learning><deep-learning><deep-rl><dqn><deep-neural-networks>
Title: How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
UserDisplayName: Reese
ThreadText: # How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
  
  (Asked by: username_0 on 2023-07-26)
  
  I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
  However, there is a constraint that one task should be served by only one server.How should I design the action space?
  
  So when I design the decision-making action space,I don't know which design is better:
  
  ```
          self.action_space = spaces.MultiDiscrete([2]*(M*N)) #action space: [a_11,a_12,...,a_1N,a_M1,a_M2,...,a_MN]  M*N discrete vector,a_mn in [0,1]
          self.action_space = spaces.MultiDiscrete([M]*N) #action space: [a_1,a_2,...,a_N] N discrete vector, a_n in [0,M-1]
  
  ```
  
  In first way, the variable a\_mn stands for if the n-th task is offloaded to m-th server or not. While in the second way,a\_n represents which server should be chosen by the n-th task.
  
  If the first way is better, how can I implement the constraints mentioned before, since there is lots of infeasible actions in the action space?
  
  And I was confused in both situation if it is possible to use the DQN to produce the Q value of all action from this action space?
  Thank a lot
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl, dqn, deep-neural-networks
  
  > **Comments:**
  > **username_1**: maybe https://towardsdatascience.com/action-masking-with-rllib-5e4bec5e7505?  
  > **username_2**: It may be easier to allow infeasible actions to exist in your problem. Instead, set them to transition to a terminal state returning a highly negative reward. This should prevent such actions from being selected at evaluation, however, it depends on whether it would be ok to select these sort of actions in training.  
  > **username_0**: Thanks a lot. Do you mean the first way is better, even there is lots of actions infeasible in the first action space?  
  
  ---
  
  ## 0 answers
  
  