Answers.Body: <p>In general, if you have a composite function <span class="math-container">$h(x) = g(f(x))$</span>, then <span class="math-container">$\frac{dh}{dx} = \frac{d g}{df} \frac{d f}{dx}$</span>. In your case, the function to differentiate is</p>
  <p><span class="math-container">$$L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim  U(D)} \left[ \left(r+\gamma \max_{a '} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}) \right)^2 \right]$$</span></p>
  <p>So, we want to calculate <span class="math-container">$\nabla_{\theta_i} L_{i}(\theta_{i})$</span>, which is equal to</p>
  <p><span class="math-container">$$\nabla_{\theta_i}  \mathbb{E}_{(s,a,r,s') \sim  U(D)} \left[ \left(r+\gamma \max_{a '} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}) \right)^2 \right] \label{2} \tag{2} $$</span></p>
  <p>For clarity, let's ignore the iteration number <span class="math-container">$i$</span> in \ref{2}, which can thus more simply be written as</p>
  <p><span class="math-container">$$\nabla_{\theta}  \mathbb{E}_{(s,a,r,s') \sim  U(D)} \left[ \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta ) \right)^2 \right] \label{3} \tag{3} $$</span></p>
  <p>The <a href="https://stats.stackexchange.com/q/72613/82135">subscript of the expected value operator</a> <span class="math-container">$\mathbf{e}=(s,a,r,s') \sim  U(D)$</span> means that the expected value is being taken with respect to the <a href="https://en.wikipedia.org/wiki/Multivariate_random_variable" rel="nofollow noreferrer"><em>multivariate</em> random variable</a>, <span class="math-container">$\mathbf{E}$</span> (for <em>experience</em>), whose values (or <a href="https://en.wikipedia.org/wiki/Realization_(probability)" rel="nofollow noreferrer">realizations</a>) are <span class="math-container">$\mathbf{e}=(s,a,r,s')$</span>, and that follows the distribution <span class="math-container">$U(D)$</span> (a uniform distribution), that is, <span class="math-container">$\mathbf{e}=(s,a,r,s')$</span> are uniformly drawn from the experience replay buffer, <span class="math-container">$D$</span> (or <span class="math-container">$\mathbf{E} \sim U(D)$</span>). However, let's ignore or omit this subscript for now (because there are no other random variables in \ref{3}, given that <span class="math-container">$\gamma$</span> and <span class="math-container">$a'$</span> should not be random variables, thus it should not be ambiguous with respect to which random variable the expectation is being calculated), so \ref{3} can be written as</p>
  <p><span class="math-container">$$\nabla_{\theta}  \mathbb{E} \left[ \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta ) \right)^2 \right] \label{4} \tag{4} $$</span></p>
  <p>Now, recall that, in the case of a discrete random variable, the expected value is a weighted sum. In the case of a continuous random variable, it is an integral. So, if <span class="math-container">$\mathbf{E}$</span> is a continuous random variable, then the <em>expectation</em> \ref{4} can be expanded to</p>
  <p><span class="math-container">$$
  \int_{\mathbb{D}} {\left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)}^2 f(\mathbf{e}) d\mathbf{e}
  $$</span></p>
  <p>where <span class="math-container">$f$</span> is the density function associated with <span class="math-container">$\mathbf{E}$</span> and <span class="math-container">$\mathbb{D}$</span> is the domain of the random variable <span class="math-container">$\mathbf{E}$</span>.</p>
  <p>The derivative of an integral can be calculated with the <a href="https://en.wikipedia.org/wiki/Leibniz_integral_rule" rel="nofollow noreferrer">Leibniz integral rule</a>. In the case the bounds of the integration are constants (<span class="math-container">$a$</span> and <span class="math-container">$b$</span>), then the Leibniz integral rule reduces to</p>
  <p><span class="math-container">$${\displaystyle {\frac {d}{dx}}\left(\int _{a}^{b}f(x,t)\,dt\right)=\int _{a}^{b}{\frac {\partial }{\partial x}}f(x,t)\,dt.}$$</span></p>
  <p>Observe that the derivative is taken with respect to the variable <span class="math-container">$x$</span>, while the integration is taken with respect to variable <span class="math-container">$t$</span>.</p>
  <p>In our case, the domain of integration, <span class="math-container">$\mathbb{D}$</span>, is constant because it only represents all experience in the dataset, <span class="math-container">$\mathcal{D}$</span>. Therefore, the gradient in \ref{4} can be written as</p>
  <p><span class="math-container">\begin{align}
  \nabla_{\theta} \int_{\mathbb{D}} \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)^2 f(\mathbf{e}) d\mathbf{e} 
  &amp;= \\
  \int_{\mathbb{D}} \nabla_{\theta} \left( \left( r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)^2  f(\mathbf{e}) \right) d\mathbf{e}
  &amp;=\\
  \mathbb{E} \left[ \nabla_{\theta} { \left(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta) \right)}^2 \right] \label{5}\tag{5}
  \end{align}</span></p>
  <p>Recall that the derivative of <span class="math-container">$f(x)=x^2$</span> is <span class="math-container">$f'(x)=2x$</span> and that the derivative of a constant is zero. Note now that the only term in \ref{5} that contains <span class="math-container">$\theta$</span> is <span class="math-container">$Q(s,a; \theta)$</span>, so all other terms are constant with respect to <span class="math-container">$\theta$</span>. Hence, \ref{5} can be written as</p>
  <p><span class="math-container">\begin{align}
  \mathbb{E} \left[ \nabla_{\theta} {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}^2 \right] 
  &amp;=\\
  \mathbb{E} \left[ 2 {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))} \nabla_{\theta} \left( {r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta)} \right) \right] 
  &amp;=\\
  \mathbb{E} \left[ 2 {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}  \left( {\nabla_{\theta} r + \nabla_{\theta} \gamma \max_{a'} Q(s',a',\theta^{-}) - \nabla_{\theta} Q(s,a; \theta)} \right) \right] 
  &amp;=\\
  \mathbb{E} \left[ - 2 {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))} {\nabla_{\theta} Q(s,a; \theta)} \right] 
  &amp;=\\
  -2 \mathbb{E} \left[ {(r + \gamma \max_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}{\nabla_{\theta} Q(s,a; \theta)} \right] 
  \end{align}</span></p>
  <p>The <span class="math-container">$-2$</span> disappears because it can be absorbed by the learning rate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-07T17:24:21.510Z
Answers.LastActivityDate: 2021-01-18T00:59:18.373Z
Answers.LastEditDate: 2021-01-18T00:59:18.373Z
Answers.UserDisplayName: nbro
Body: <p>In the <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">original DQN paper</a>, page 1, the loss function of the DQN is</p>
  
  <p><span class="math-container">$$
  L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim  U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2]
  $$</span></p>
  
  <p>whose gradient is presented (on page 7)</p>
  
  <p><span class="math-container">$$\nabla_{\theta_i} L_i(\theta_i) = \mathbb{E}_{s,a,r,s'} [(r+\gamma \max_{a'}Q(s',a';\theta_i^-) - Q(s,a;\theta_i))\nabla_{\theta_i}Q(s,a;\theta_i)] $$</span></p>
  
  <p>But why is there no minus (-) sign if <span class="math-container">$-Q(s,a;\theta_i)$</span> is parameterized by <span class="math-container">$\theta_i$</span> and why is the 2 from power gone?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-14T10:15:52.407Z
Comments.Text: Because it is wrong and there should be a minus, period.
Comments.UserDisplayName: Kuo
Tags: <reinforcement-learning><dqn><deep-rl><gradient-descent><gradient>
Title: How is the gradient of the loss function in DQN derived?
UserDisplayName: Dimitris Monroe
ThreadText: # How is the gradient of the loss function in DQN derived?
  
  (Asked by: username_0 on 2019-09-07)
  
  In the [original DQN paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf), page 1, the loss function of the DQN is
  
  $$
  L\_{i}(\theta\_{i}) = \mathbb{E}\_{(s,a,r,s') \sim U(D)} [(r+\gamma \max\_{a'} Q(s',a',\theta\_{i}^{-}) - Q(s,a;\theta\_{i}))^2]
  $$
  
  whose gradient is presented (on page 7)
  
  $$\nabla\_{\theta\_i} L\_i(\theta\_i) = \mathbb{E}\_{s,a,r,s'} [(r+\gamma \max\_{a'}Q(s',a';\theta\_i^-) - Q(s,a;\theta\_i))\nabla\_{\theta\_i}Q(s,a;\theta\_i)] $$
  
  But why is there no minus (-) sign if $-Q(s,a;\theta\_i)$ is parameterized by $\theta\_i$ and why is the 2 from power gone?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, gradient-descent, gradient
  
  > **Comments:**
  > **username_1**: Because it is wrong and there should be a minus, period.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-07. Score: 2 | Accepted answer
  
  In general, if you have a composite function $h(x) = g(f(x))$, then $\frac{dh}{dx} = \frac{d g}{df} \frac{d f}{dx}$. In your case, the function to differentiate is
  
  $$L\_{i}(\theta\_{i}) = \mathbb{E}\_{(s,a,r,s') \sim U(D)} \left[ \left(r+\gamma \max\_{a '} Q(s',a',\theta\_{i}^{-}) - Q(s,a;\theta\_{i}) \right)^2 \right]$$
  
  So, we want to calculate $\nabla\_{\theta\_i} L\_{i}(\theta\_{i})$, which is equal to
  
  $$\nabla\_{\theta\_i} \mathbb{E}\_{(s,a,r,s') \sim U(D)} \left[ \left(r+\gamma \max\_{a '} Q(s',a',\theta\_{i}^{-}) - Q(s,a;\theta\_{i}) \right)^2 \right] \label{2} \tag{2} $$
  
  For clarity, let's ignore the iteration number $i$ in \ref{2}, which can thus more simply be written as
  
  $$\nabla\_{\theta} \mathbb{E}\_{(s,a,r,s') \sim U(D)} \left[ \left(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta ) \right)^2 \right] \label{3} \tag{3} $$
  
  The [subscript of the expected value operator](https://stats.stackexchange.com/q/72613/82135) $\mathbf{e}=(s,a,r,s') \sim U(D)$ means that the expected value is being taken with respect to the [*multivariate* random variable](https://en.wikipedia.org/wiki/Multivariate_random_variable), $\mathbf{E}$ (for *experience*), whose values (or [realizations](https://en.wikipedia.org/wiki/Realization_(probability))) are $\mathbf{e}=(s,a,r,s')$, and that follows the distribution $U(D)$ (a uniform distribution), that is, $\mathbf{e}=(s,a,r,s')$ are uniformly drawn from the experience replay buffer, $D$ (or $\mathbf{E} \sim U(D)$). However, let's ignore or omit this subscript for now (because there are no other random variables in \ref{3}, given that $\gamma$ and $a'$ should not be random variables, thus it should not be ambiguous with respect to which random variable the expectation is being calculated), so \ref{3} can be written as
  
  $$\nabla\_{\theta} \mathbb{E} \left[ \left(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta ) \right)^2 \right] \label{4} \tag{4} $$
  
  Now, recall that, in the case of a discrete random variable, the expected value is a weighted sum. In the case of a continuous random variable, it is an integral. So, if $\mathbf{E}$ is a continuous random variable, then the *expectation* \ref{4} can be expanded to
  
  $$
  \int\_{\mathbb{D}} {\left(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)}^2 f(\mathbf{e}) d\mathbf{e}
  $$
  
  where $f$ is the density function associated with $\mathbf{E}$ and $\mathbb{D}$ is the domain of the random variable $\mathbf{E}$.
  
  The derivative of an integral can be calculated with the [Leibniz integral rule](https://en.wikipedia.org/wiki/Leibniz_integral_rule). In the case the bounds of the integration are constants ($a$ and $b$), then the Leibniz integral rule reduces to
  
  $${\displaystyle {\frac {d}{dx}}\left(\int \_{a}^{b}f(x,t)\,dt\right)=\int \_{a}^{b}{\frac {\partial }{\partial x}}f(x,t)\,dt.}$$
  
  Observe that the derivative is taken with respect to the variable $x$, while the integration is taken with respect to variable $t$.
  
  In our case, the domain of integration, $\mathbb{D}$, is constant because it only represents all experience in the dataset, $\mathcal{D}$. Therefore, the gradient in \ref{4} can be written as
  
  \begin{align}
  \nabla\_{\theta} \int\_{\mathbb{D}} \left(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)^2 f(\mathbf{e}) d\mathbf{e}
  &= \\
  \int\_{\mathbb{D}} \nabla\_{\theta} \left( \left( r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta )\right)^2 f(\mathbf{e}) \right) d\mathbf{e}
  &=\\
  \mathbb{E} \left[ \nabla\_{\theta} { \left(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta) \right)}^2 \right] \label{5}\tag{5}
  \end{align}
  
  Recall that the derivative of $f(x)=x^2$ is $f'(x)=2x$ and that the derivative of a constant is zero. Note now that the only term in \ref{5} that contains $\theta$ is $Q(s,a; \theta)$, so all other terms are constant with respect to $\theta$. Hence, \ref{5} can be written as
  
  \begin{align}
  \mathbb{E} \left[ \nabla\_{\theta} {(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}^2 \right]
  &=\\
  \mathbb{E} \left[ 2 {(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))} \nabla\_{\theta} \left( {r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta)} \right) \right]
  &=\\
  \mathbb{E} \left[ 2 {(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))} \left( {\nabla\_{\theta} r + \nabla\_{\theta} \gamma \max\_{a'} Q(s',a',\theta^{-}) - \nabla\_{\theta} Q(s,a; \theta)} \right) \right]
  &=\\
  \mathbb{E} \left[ - 2 {(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))} {\nabla\_{\theta} Q(s,a; \theta)} \right]
  &=\\
  -2 \mathbb{E} \left[ {(r + \gamma \max\_{a'} Q(s',a',\theta^{-}) - Q(s,a; \theta))}{\nabla\_{\theta} Q(s,a; \theta)} \right]
  \end{align}
  
  The $-2$ disappears because it can be absorbed by the learning rate.
  
  
  ---
  
  