Answers.Body: <p>You calculate the max by calculating your estimates for all possible actions for the next state, and taking the highest value.</p>
  
  <p>The details depend a little on your neural network architecture:</p>
  
  <ul>
  <li><p>If you have a network that takes the state vector as input and outputs all possible action values <span class="math-container">$(\hat{q}(s,a_0), \hat{q}(s,a_1), \hat{q}(s,a_2) ...)$</span>, then you can run it once forward with the <code>next_state</code> as input to getthe array <span class="math-container">$(\hat{q}(s',a_0), \hat{q}(s',a_1), \hat{q}(s',a_2) ...)$</span>, and take the maximum element (you don't need to care which action <span class="math-container">$a'$</span> caused it). You will then have the problem that you now have a loss for <span class="math-container">$Q(s, a, \mathbf{w})$</span> for the single action <span class="math-container">$a$</span> just taken, but no data any of the alternative actions. The loss for these alternative actions needs to be set to zero - if you are training the NN using a normal supervised learning approach, that means you need to keep the full output of the network that you ran forward to calculate <span class="math-container">$Q(s, a, \mathbf{w})$</span> then substitute in this new estimated value against the action <span class="math-container">$a$</span> and train using this modified vector.</p></li>
  <li><p>If you have a network that takes the state vector and action combined as input and outputs a single estimate <span class="math-container">$\hat{q}(s,a)$</span>, then you have to run that network once for each possible action from the next state and take the maximum value. You would typically do this as a small batch prediction for better performance. In this case your training data is simple to construct as you only have the loss for and train the network against one state/action combination. </p></li>
  </ul>
  
  <p>Overall the first option (all action values at once) is usually a lot more efficient, but slightly more complex to code the training routine.</p>
  
  <blockquote>
    <p>do you really need to hold always an older version of the network. If yes why and how old should it be?</p>
  </blockquote>
  
  <p>You don't have to, but it is highly advisable to have this target network (so called because it helps generate your TD Target values), because Q learning using neural networks is often unstable. This is due to the bootstrap calculations where estimates are based on other estimates plus a little bit of observed data at each step. There is a strong possibility for runaway feedback due to training a neural network on something that includes its own output.</p>
  
  <p>How old should it be? That's sadly a hyper-parameter of the architecture that you will need to establish through experiment on each new problem. I have worked with maximum age values from 100 to 10000 in my own simple experiments. Note this is not usually a rolling age - you don't keep 1000 copies of the network weights. Just keep one frozen copy, and after N steps replace it with a copy of the most recent one.</p>
  
  <p>One alternative to this copy/freeze/copy approach is to update the target network towards the learning network on every step by a small factor. E.g. <span class="math-container">$\mathbf{w}^{\_} = (1 - \beta)\mathbf{w}^{\_} + \beta \mathbf{w}$</span> where <span class="math-container">$\beta$</span> might be <span class="math-container">$0.001$</span></p>
  
  <p>In addition, you should be using experience replay for training data, and not training directly online. The combination of experience replay and using a frozen or slowly adapting target network makes a large difference to the stability of deep Q learning in practice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T19:17:29.007Z
Answers.Text: I'd describe the need for target networks being due to the "moving target" problem that you otherwise get, rather than numeric stability. Numeric stability can easily be confused with problems/inaccuracies related to floating point arithmetic, which is not really the problem addressed by target networks (although I suppose in some cases the moving target instability may get so severe that reaching the realms of NaN becomes a symptom... but that's not the only possible symptom I think?)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T19:44:55.643Z
Answers.Text: So what wrong with the Brain.class (I did exactly this except experience replay)
  https://github.com/SuchtyTV/RLearningBird/blob/master/src/main/java/rlgame/Brain.java
Answers.UserDisplayName: TVSuchty
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T20:25:23.253Z
Answers.Text: @TVSuchty: Sorry I am not going to try and debug your project. I don't work in Java so it would take me too long and I may miss any obvious problem.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T20:29:19.060Z
Answers.Text: Could you help me if i translate it into pseudocode? or python?
Answers.UserDisplayName: TVSuchty
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T20:31:53.287Z
Answers.Text: @TVSuchty: Sorry no. If you can write a question about some part of the code that you are having trouble with, then I or someone else here might be able to help you with an answer. One thing you could do is write out a brief high-level pseudo-code for what your Brain class is doing, and ask if that is correct in a question, or if you have implemented it and it is not working, ask about the problems you are having. There are already many examples of working DQN in Python available though if you search . . .
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T18:32:17.463Z
Answers.LastActivityDate: 2019-01-05T19:24:38.983Z
Answers.LastEditDate: 2019-01-05T19:24:38.983Z
Answers.UserDisplayName: Neil Slater
Body: <p>Consider the following loss function</p>
  
  <p><span class="math-container">$$
  L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2
  $$</span></p>
  
  <p>where <span class="math-container">$Q(s, a, \mathbf{w^-})$</span> and <span class="math-container">$Q(s, a, \mathbf{w})$</span> are represented as neural networks, where <span class="math-container">$w^-$</span> and <span class="math-container">$w$</span> are the corresponding weights.</p>
  
  <p>But how do you calculate <span class="math-container">$max_{a'} Q(s', a', \mathbf{w^-})$</span>? Do you really need to hold always an older version of the network? If yes, why and how old should it be?</p>
  
Tags: <reinforcement-learning><q-learning><objective-functions><deep-rl>
Title: How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?
UserDisplayName: TVSuchty
ThreadText: # How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?
  
  (Asked by: username_0 on 2019-01-05)
  
  Consider the following loss function
  
  $$
  L(\mathbf{w}) = [(r + \gamma max\_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2
  $$
  
  where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are represented as neural networks, where $w^-$ and $w$ are the corresponding weights.
  
  But how do you calculate $max\_{a'} Q(s', a', \mathbf{w^-})$? Do you really need to hold always an older version of the network? If yes, why and how old should it be?
  
  **Tags:** reinforcement-learning, q-learning, objective-functions, deep-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-05. Score: 1
  
  You calculate the max by calculating your estimates for all possible actions for the next state, and taking the highest value.
  
  The details depend a little on your neural network architecture:
  
  * If you have a network that takes the state vector as input and outputs all possible action values $(\hat{q}(s,a\_0), \hat{q}(s,a\_1), \hat{q}(s,a\_2) ...)$, then you can run it once forward with the `next_state` as input to getthe array $(\hat{q}(s',a\_0), \hat{q}(s',a\_1), \hat{q}(s',a\_2) ...)$, and take the maximum element (you don't need to care which action $a'$ caused it). You will then have the problem that you now have a loss for $Q(s, a, \mathbf{w})$ for the single action $a$ just taken, but no data any of the alternative actions. The loss for these alternative actions needs to be set to zero - if you are training the NN using a normal supervised learning approach, that means you need to keep the full output of the network that you ran forward to calculate $Q(s, a, \mathbf{w})$ then substitute in this new estimated value against the action $a$ and train using this modified vector.
  * If you have a network that takes the state vector and action combined as input and outputs a single estimate $\hat{q}(s,a)$, then you have to run that network once for each possible action from the next state and take the maximum value. You would typically do this as a small batch prediction for better performance. In this case your training data is simple to construct as you only have the loss for and train the network against one state/action combination.
  
  Overall the first option (all action values at once) is usually a lot more efficient, but slightly more complex to code the training routine.
  
  > do you really need to hold always an older version of the network. If yes why and how old should it be?
  
  You don't have to, but it is highly advisable to have this target network (so called because it helps generate your TD Target values), because Q learning using neural networks is often unstable. This is due to the bootstrap calculations where estimates are based on other estimates plus a little bit of observed data at each step. There is a strong possibility for runaway feedback due to training a neural network on something that includes its own output.
  
  How old should it be? That's sadly a hyper-parameter of the architecture that you will need to establish through experiment on each new problem. I have worked with maximum age values from 100 to 10000 in my own simple experiments. Note this is not usually a rolling age - you don't keep 1000 copies of the network weights. Just keep one frozen copy, and after N steps replace it with a copy of the most recent one.
  
  One alternative to this copy/freeze/copy approach is to update the target network towards the learning network on every step by a small factor. E.g. $\mathbf{w}^{\\_} = (1 - \beta)\mathbf{w}^{\\_} + \beta \mathbf{w}$ where $\beta$ might be $0.001$
  
  In addition, you should be using experience replay for training data, and not training directly online. The combination of experience replay and using a frozen or slowly adapting target network makes a large difference to the stability of deep Q learning in practice.
  
  > **Comments:**
  > **username_2**: I'd describe the need for target networks being due to the "moving target" problem that you otherwise get, rather than numeric stability. Numeric stability can easily be confused with problems/inaccuracies related to floating point arithmetic, which is not really the problem addressed by target networks (although I suppose in some cases the moving target instability may get so severe that reaching the realms of NaN becomes a symptom... but that's not the only possible symptom I think?)  
  > **username_0**: So what wrong with the Brain.class (I did exactly this except experience replay) https://github.com/SuchtyTV/RLearningBird/blob/master/src/main/java/rlgame/Brain.java  
  > **username_1**: @TVSuchty: Sorry I am not going to try and debug your project. I don't work in Java so it would take me too long and I may miss any obvious problem.  
  > **username_0**: Could you help me if i translate it into pseudocode? or python?  
  > **username_1**: @TVSuchty: Sorry no. If you can write a question about some part of the code that you are having trouble with, then I or someone else here might be able to help you with an answer. One thing you could do is write out a brief high-level pseudo-code for what your Brain class is doing, and ask if that is correct in a question, or if you have implemented it and it is not working, ask about the problems you are having. There are already many examples of working DQN in Python available though if you search . . .  
  
  ---
  
  