Answers.Body: <p>As you say, the output of a <span class="math-container">$Q$</span> network is typically a value for all actions of the given state. Let us call this output <span class="math-container">$\mathbf{x} \in \mathbb{R}^{|\mathcal{A}|}$</span>. To train your network using the squared bellman error you need first calculate the scalar target <span class="math-container">$y = r(s, a) + \max_a Q(s', a)$</span>. Then, to train the network we take a vector <span class="math-container">$\mathbf{x'} = \mathbf{x}$</span> and change the <span class="math-container">$a$</span>th element of it to be equal to <span class="math-container">$y$</span>, where <span class="math-container">$a$</span> is the action you took in state <span class="math-container">$s$</span>; call this modified vector <span class="math-container">$\mathbf{x'}_a$</span>. We calculate the loss <span class="math-container">$\mathcal{L}(\mathbf{x}, \mathbf{x'}_a)$</span> and back propagate through this to update the parameters of our network.</p>
  <p>Note that when we use <span class="math-container">$Q$</span> to calculate <span class="math-container">$y$</span> we typically use some form of target network; this can be a copy of <span class="math-container">$Q$</span> where the parameters are only updated every <span class="math-container">$i$</span>th update or a network whose weights are updated using a polyak average with the main networks weights after every update.</p>
  <p>Judging by your code it looks as though your action selection is what might be causing you some problems. As far as I can tell you're always acting greedily with respect to your <span class="math-container">$Q$</span>-function. You should be looking to act <span class="math-container">$\epsilon$</span>-greedily, i.e. with probability <span class="math-container">$\epsilon$</span> take a random action and act greedily otherwise. Typically you start with <span class="math-container">$\epsilon=1$</span> and decay it each time a random action is taken down to some small value such as 0.05.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-07T16:46:04.070Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/119470/discussion-on-answer-by-david-ireland-what-is-the-target-output-for-updating-a-d).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-02T22:09:51.500Z
Answers.LastActivityDate: 2021-02-02T22:15:28.083Z
Answers.LastEditDate: 2021-02-02T22:15:28.083Z
Answers.UserDisplayName: David
Answers.Body: <p>There are a couple ways you can define the architecture of a DQN. The most common way of doing it is by taking in the states and outputting the value function of all possible actions - this leads to a DQN with multiple outputs. The other, less efficient way, includes taking in an state-action as input and outputting a single real value - this approach is typically avoided since we need to run the model multiple times to get estimates for different actions.</p>
  <p>The replay buffer is used to store <span class="math-container">$(S,A,R,S')$</span> transitions as encountered using your <span class="math-container">$\epsilon$</span>-soft policy. We sample one of these transitions from the replay buffer and calculate an estimate of the value function for <span class="math-container">$(S,A)$</span> i.e <span class="math-container">$\hat Q(S,A,\theta)$</span> and then we calculate a target as follows. <span class="math-container">$$target =R+\max_\limits{a'}\hat Q(S',a',\theta^-)$$</span></p>
  <p>Assuming you use the first model, you can then use a Squared error loss function, defined as follows, and modify your parameter as a function of that</p>
  <p><span class="math-container">$$L(\theta) = (target-\hat Q(S,A,\theta))^2$$</span></p>
  <p>Assuming for now the target is fixed (I'll explain this in a minute), only <span class="math-container">$Q(S,A,\theta)$</span> is a function of <span class="math-container">$\theta$</span> in the loss function. <span class="math-container">$Q(S,A,\theta)$</span> corresponds to one output node of your DQN and therefore, as you've already highlighted, when carrying out EBP the parameters are updated such that we make the value of this one node tend to the specified target.</p>
  <p>This is just how Q-learning works, we use samples generated by the behaviour policy to create <span class="math-container">$L(\theta)$</span> and then tweak the parameters to minimise the cost. As we do this for more and more samples the network hopefully figures out a way that accommodates for every sample it's been trained on so far (with more emphasis on the most recent samples).</p>
  <p><strong>As to your issue, are you sure you're training on multiple different samples and not just a specific one? it may just be a bug you've overseen.</strong></p>
  <hr />
  <p><strong>Explaining  <span class="math-container">$\theta^-$</span></strong></p>
  <p>I used a slightly different notation, <span class="math-container">$\theta^-$</span>, for the parameters used to generate the bootstrapped estimate, <span class="math-container">$\max_\limits{a'}\hat Q(S',a,\theta^-)$</span>. <span class="math-container">$\theta^-$</span> is only matched to <span class="math-container">$\theta$</span> every <span class="math-container">$n^{th}$</span> step because we want to keep the target constant as much as possible. The reason for this is because Q-learning does not necessarily converge when using neural networks partly due to bootstrapping which can cause a divergence of optimisation because of state generalisation. By using this <span class="math-container">$\theta^-$</span> we help prevent things like this from happening.</p>
  <p>Ultimately the idea of the replay buffer and the fixed parameter for bootstrapping are to try to convert the RL problem into a supervised learning problem because we know much more about how to deal with supervised learning problems when using DNNs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-03T09:42:59.670Z
Answers.LastActivityDate: 2021-02-03T10:51:20.753Z
Answers.LastEditDate: 2021-02-03T10:51:20.753Z
Answers.UserDisplayName: quest ions
Body: <p>I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.</p>
  <p>The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single <span class="math-container">$Q(s, a)$</span>. However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?</p>
  <p>I've been trying to get it to work with something like the following:</p>
  <pre><code>q_values = model(state)
  action = argmax(q_values)
  next_state = env.step(state, action)
  next_q_values = model(next_state)
  max_next_q = max(next_q_values)
  
  target_q_values = q_values
  target_q_values[action] = reward(next_state) + gamma * max_next_q
  </code></pre>
  <p>The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)</p>
  <p>What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?</p>
  
Tags: <neural-networks><reinforcement-learning><q-learning><dqn><deep-rl>
Title: What is the target output for updating a Deep Q Network
UserDisplayName: Kricket
ThreadText: # What is the target output for updating a Deep Q Network
  
  (Asked by: username_0 on 2021-02-02)
  
  I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.
  The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single $Q(s, a)$. However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?
  I've been trying to get it to work with something like the following:
  q_values = model(state)
  action = argmax(q_values)
  next_state = env.step(state, action)
  next_q_values = model(next_state)
  max_next_q = max(next_q_values)
  
  target_q_values = q_values
  target_q_values[action] = reward(next_state) + gamma * max_next_q
  
  The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)
  What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?
  
  **Tags:** neural-networks, reinforcement-learning, q-learning, dqn, deep-rl
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-02-02. Score: 3 | Accepted answer
  
  As you say, the output of a $Q$ network is typically a value for all actions of the given state. Let us call this output $\mathbf{x} \in \mathbb{R}^{|\mathcal{A}|}$. To train your network using the squared bellman error you need first calculate the scalar target $y = r(s, a) + \max\_a Q(s', a)$. Then, to train the network we take a vector $\mathbf{x'} = \mathbf{x}$ and change the $a$th element of it to be equal to $y$, where $a$ is the action you took in state $s$; call this modified vector $\mathbf{x'}\_a$. We calculate the loss $\mathcal{L}(\mathbf{x}, \mathbf{x'}\_a)$ and back propagate through this to update the parameters of our network.
  
  Note that when we use $Q$ to calculate $y$ we typically use some form of target network; this can be a copy of $Q$ where the parameters are only updated every $i$th update or a network whose weights are updated using a polyak average with the main networks weights after every update.
  
  Judging by your code it looks as though your action selection is what might be causing you some problems. As far as I can tell you're always acting greedily with respect to your $Q$-function. You should be looking to act $\epsilon$-greedily, i.e. with probability $\epsilon$ take a random action and act greedily otherwise. Typically you start with $\epsilon=1$ and decay it each time a random action is taken down to some small value such as 0.05.
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/119470/discussion-on-answer-by-david-ireland-what-is-the-target-output-for-updating-a-d).  
  
  ---
  
  ### Answer by username_3 on 2021-02-03. Score: 1
  
  There are a couple ways you can define the architecture of a DQN. The most common way of doing it is by taking in the states and outputting the value function of all possible actions - this leads to a DQN with multiple outputs. The other, less efficient way, includes taking in an state-action as input and outputting a single real value - this approach is typically avoided since we need to run the model multiple times to get estimates for different actions.
  
  The replay buffer is used to store $(S,A,R,S')$ transitions as encountered using your $\epsilon$-soft policy. We sample one of these transitions from the replay buffer and calculate an estimate of the value function for $(S,A)$ i.e $\hat Q(S,A,\theta)$ and then we calculate a target as follows. $$target =R+\max\_\limits{a'}\hat Q(S',a',\theta^-)$$
  
  Assuming you use the first model, you can then use a Squared error loss function, defined as follows, and modify your parameter as a function of that
  
  $$L(\theta) = (target-\hat Q(S,A,\theta))^2$$
  
  Assuming for now the target is fixed (I'll explain this in a minute), only $Q(S,A,\theta)$ is a function of $\theta$ in the loss function. $Q(S,A,\theta)$ corresponds to one output node of your DQN and therefore, as you've already highlighted, when carrying out EBP the parameters are updated such that we make the value of this one node tend to the specified target.
  
  This is just how Q-learning works, we use samples generated by the behaviour policy to create $L(\theta)$ and then tweak the parameters to minimise the cost. As we do this for more and more samples the network hopefully figures out a way that accommodates for every sample it's been trained on so far (with more emphasis on the most recent samples).
  
  **As to your issue, are you sure you're training on multiple different samples and not just a specific one? it may just be a bug you've overseen.**
  
  ---
  
  **Explaining $\theta^-$**
  
  I used a slightly different notation, $\theta^-$, for the parameters used to generate the bootstrapped estimate, $\max\_\limits{a'}\hat Q(S',a,\theta^-)$. $\theta^-$ is only matched to $\theta$ every $n^{th}$ step because we want to keep the target constant as much as possible. The reason for this is because Q-learning does not necessarily converge when using neural networks partly due to bootstrapping which can cause a divergence of optimisation because of state generalisation. By using this $\theta^-$ we help prevent things like this from happening.
  
  Ultimately the idea of the replay buffer and the fixed parameter for bootstrapping are to try to convert the RL problem into a supervised learning problem because we know much more about how to deal with supervised learning problems when using DNNs.
  
  
  ---
  
  