Answers.Body: <h2>1) Is there any way to set the initial Q-values for the actions?</h2>
  <p>You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network weights directly, at least. That would defeat the purpose of using backpropagation to optimize the weights and find the optimal parameters and Q-values.</p>
  <h2>2) Is there any way to limit the set of valid actions per state?</h2>
  <p>Again, not directly through the network. That would imply different network architectures for different states, which would imply optimizing several different networks. Unless you used some convoluted approach like Net2Net to keep the networks synced.</p>
  <h2>However, you can bypass the network itself.</h2>
  <p>You can, however, on top of the network, filter its outputs. Let's say you have a set of states, and in state X, only 2 out of all 5 actions are valid. When getting the Q-values for all the actions in DQN, you can detect if you're in state X, and instead of choosing the greedy action out of all actions, choose the greedy action out of all <strong>valid</strong> actions. Better yet, have your environment just send a list of all valid actions for whatever state you're in. Makes this more general, as it removes your need for discrete states, having lists of valid actions for each state, etc.</p>
  <h2>You can also really just ignore invalid actions.</h2>
  <p>You claimed</p>
  <blockquote>
  <p>However, ignoring them would imply returning the same state and a 0 reward, the best possible outcome, which is clearly not the case.</p>
  </blockquote>
  <p>If your environment has <strong>no time limit</strong>, then the best action really is to stand still and avoid wasting resources. However, if there is a time-limit, eventually the agent should receive some sort of penalty for not completing the task. Otherwise, if there is no time limit, you can also give him an award for completing the task. Imagining you have 10 resources, instead of having your reward interval be from <code>[-10; 0]</code>, shift it to <code>[0, 10]</code> by awarding 10 points at the end of the game. This gives your agent the motivation to complete the task instead of doing useless actions, and discards your need for action-filtering at all.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-29T12:30:11.967Z
Answers.Text: I do not quite understand why initially setting the network weights would defeat the backpropagation purpose? Why is it unfeasible to use quick, greedy heuristics to estimate the value of a certain state, can you elaborate? 
  Setting an end reward motivates the agent to reach the end, but it would still need the negative rewards to keep it from using resources it does not need, right? There is no time limit: I do not care if it takes longer to complete a task, as long as it uses as few resources as possible.
Answers.UserDisplayName: AlexGuevara
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-29T12:46:29.027Z
Answers.Text: You can use some heuristics to start with a network whose weights are not completely random. It is called transfer learning. But it's hard to do because it implies you knowing the values somehow (usually, through previous learning stages). You also cannot just set some specific values and expect others to remain unchanged. As you adjust weights to get some actions to -inf, other action's will get very strange reward values, which will most likely disturb training. Once you start optimizing the network, when you explore, backprop will just optimize the weights and set everything back.
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-29T12:47:40.883Z
Answers.Text: By setting an end reward, you just motivate the agent to finish the task (since he gets the best reward out of finishing it). You still need the negative rewards, of course, to make him avoid wasting resources. With penalties and an end-game reward, you get the best of both worlds.
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-24T12:25:47.797Z
Answers.Text: @BlueMoon93, I don't understand how changinng reward interval from [-10,0] to [0,+10] would improve learning? In the end, it is just shifted. In the first case the agent would be motivated not to do stupid actions in order to get closer to 0...
Answers.UserDisplayName: d56
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-29T11:02:05.533Z
Answers.LastActivityDate: 2017-05-29T11:08:27.723Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: BlueMoon93
Body: <p>I created an OpenAI Gym environment, and I would like to check the performance of the agent from <a href="https://blog.openai.com/openai-baselines-dqn/" rel="nofollow noreferrer">OpenAI Baselines DQN approach </a> on it.
  In my environment, the best possible outcome for the agent is 0 - the robot needs zero non-necessary resources to complete a task. The goal is to minimize the need for resources: for each needed resource, there is a penalty of -1. In many states, only certain actions make physical sense. How do I deal with this?</p>
  <p>There was already a question about the handling of invalid moves on <a href="https://ai.stackexchange.com/q/2980/2444">AI StackExchange</a>, recommending to ignore the invalid moves. However, ignoring them would imply returning the same state and a 0 reward, the best possible outcome, which is clearly not the case. Setting drastic negative rewards also does not seem to work, since even promising handling paths are compromised by invalid actions and the corresponding drastic negative reward.</p>
  <p><em>What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best) or negative?</em></p>
  <p>My ideas/ questions on this for the <a href="https://blog.openai.com/openai-baselines-dqn/" rel="nofollow noreferrer">OpenAI Baselines DQN approach</a> implementation</p>
  <ol>
  <li><p>Is there any way to set the initial Q-values for the actions? I could set -infinity for the invalid actions.</p>
  </li>
  <li><p>Is there any way to limit the set of valid actions per state? When after the env.step(action) function the new state is returned, can I somehow define which actions are valid for it?</p>
  </li>
  </ol>
  
Tags: <reinforcement-learning><q-learning><dqn><implementation><reward-functions>
Title: What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?
UserDisplayName: AlexGuevara
ThreadText: # What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?
  
  (Asked by: username_0 on 2017-05-29)
  
  I created an OpenAI Gym environment, and I would like to check the performance of the agent from [OpenAI Baselines DQN approach](https://blog.openai.com/openai-baselines-dqn/)  on it.
  In my environment, the best possible outcome for the agent is 0 - the robot needs zero non-necessary resources to complete a task. The goal is to minimize the need for resources: for each needed resource, there is a penalty of -1. In many states, only certain actions make physical sense. How do I deal with this?
  
  There was already a question about the handling of invalid moves on [AI StackExchange](https://ai.stackexchange.com/q/2980/2444), recommending to ignore the invalid moves. However, ignoring them would imply returning the same state and a 0 reward, the best possible outcome, which is clearly not the case. Setting drastic negative rewards also does not seem to work, since even promising handling paths are compromised by invalid actions and the corresponding drastic negative reward.
  
  *What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best) or negative?*
  
  My ideas/ questions on this for the [OpenAI Baselines DQN approach](https://blog.openai.com/openai-baselines-dqn/) implementation
  
  1. Is there any way to set the initial Q-values for the actions? I could set -infinity for the invalid actions.
  2. Is there any way to limit the set of valid actions per state? When after the env.step(action) function the new state is returned, can I somehow define which actions are valid for it?
  
  **Tags:** reinforcement-learning, q-learning, dqn, implementation, reward-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-05-29. Score: 5 | Accepted answer
  
  ## 1) Is there any way to set the initial Q-values for the actions?
  
  You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network weights directly, at least. That would defeat the purpose of using backpropagation to optimize the weights and find the optimal parameters and Q-values.
  
  ## 2) Is there any way to limit the set of valid actions per state?
  
  Again, not directly through the network. That would imply different network architectures for different states, which would imply optimizing several different networks. Unless you used some convoluted approach like Net2Net to keep the networks synced.
  
  ## However, you can bypass the network itself.
  
  You can, however, on top of the network, filter its outputs. Let's say you have a set of states, and in state X, only 2 out of all 5 actions are valid. When getting the Q-values for all the actions in DQN, you can detect if you're in state X, and instead of choosing the greedy action out of all actions, choose the greedy action out of all **valid** actions. Better yet, have your environment just send a list of all valid actions for whatever state you're in. Makes this more general, as it removes your need for discrete states, having lists of valid actions for each state, etc.
  
  ## You can also really just ignore invalid actions.
  
  You claimed
  
  > However, ignoring them would imply returning the same state and a 0 reward, the best possible outcome, which is clearly not the case.
  
  If your environment has **no time limit**, then the best action really is to stand still and avoid wasting resources. However, if there is a time-limit, eventually the agent should receive some sort of penalty for not completing the task. Otherwise, if there is no time limit, you can also give him an award for completing the task. Imagining you have 10 resources, instead of having your reward interval be from `[-10; 0]`, shift it to `[0, 10]` by awarding 10 points at the end of the game. This gives your agent the motivation to complete the task instead of doing useless actions, and discards your need for action-filtering at all.
  
  > **Comments:**
  > **username_0**: I do not quite understand why initially setting the network weights would defeat the backpropagation purpose? Why is it unfeasible to use quick, greedy heuristics to estimate the value of a certain state, can you elaborate?  Setting an end reward motivates the agent to reach the end, but it would still need the negative rewards to keep it from using resources it does not need, right? There is no time limit: I do not care if it takes longer to complete a task, as long as it uses as few resources as possible.  
  > **username_1**: You can use some heuristics to start with a network whose weights are not completely random. It is called transfer learning. But it's hard to do because it implies you knowing the values somehow (usually, through previous learning stages). You also cannot just set some specific values and expect others to remain unchanged. As you adjust weights to get some actions to -inf, other action's will get very strange reward values, which will most likely disturb training. Once you start optimizing the network, when you explore, backprop will just optimize the weights and set everything back.  
  > **username_1**: By setting an end reward, you just motivate the agent to finish the task (since he gets the best reward out of finishing it). You still need the negative rewards, of course, to make him avoid wasting resources. With penalties and an end-game reward, you get the best of both worlds.  
  > **username_2**: @username_1, I don't understand how changinng reward interval from [-10,0] to [0,+10] would improve learning? In the end, it is just shifted. In the first case the agent would be motivated not to do stupid actions in order to get closer to 0...  
  
  ---
  
  