Answers.Body: <p>first we need to transform the distribution of the first term:</p>
  <p><span class="math-container">$\mathrm{argmin} \space \alpha \mathrm{E}_{s\sim D, a \sim D}[\frac{\mu}{\hat\pi_{\beta}}Q(s,a)] + \frac{1}{2}\mathrm{E}_{s,a \sim D}(Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a))^2   $</span></p>
  <p>then its derivate respect to Q is:</p>
  <p><span class="math-container">$\alpha\frac{\mu}{\hat \pi ^{\beta}} + Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a) = 0$</span></p>
  <p>then we have</p>
  <p><span class="math-container">$ Q(s,a) =\hat\beta^{\pi}\hat Q^k(s,a) -\alpha\frac{\mu}{\hat \pi ^{\beta}}$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T23:53:11.020Z
Answers.Text: how did you find the derivative with respect to Q?
Answers.UserDisplayName: MoneyBall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T09:16:02.843Z
Answers.LastActivityDate: 2021-12-15T09:16:02.843Z
Answers.UserDisplayName: WryHumor
Body: <p>I am looking at the paper <a href="https://arxiv.org/abs/2006.04779" rel="nofollow noreferrer">Conservative Q-Learning for Offline Reinforcement Learning</a>, but I'm not sure how they proved theorem 3.1.</p>
  <p>Here is a screenshot of theorem 3.1.</p>
  <p><a href="https://i.sstatic.net/xBfaL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xBfaL.png" alt="enter image description here" /></a></p>
  <p>In the proof of theorem 3.1</p>
  <p><a href="https://i.sstatic.net/GLIRy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GLIRy.png" alt="enter image description here" /></a></p>
  <p>they say</p>
  <blockquote>
  <p>By setting the derivative of Equation 1 to 0, we obtain the following expression</p>
  <p>...</p>
  <p><span class="math-container">$$\forall \mathbf{s}, \mathbf{a} \in \mathcal{D}, k, \quad \hat{Q}^{k+1}(\mathbf{s}, \mathbf{a})=\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})-\alpha \frac{\mu(\mathbf{a} \mid \mathbf{s})}{\hat{\pi}_{\beta}(\mathbf{a} \mid \mathbf{s})} \tag{11}\label{11}$$</span></p>
  </blockquote>
  <p>Here's equation 1 from the paper.</p>
  <p><span class="math-container">$$\hat{Q}^{k+1} \leftarrow \arg \min _{Q} \alpha \mathbb{E}_{\mathbf{s} \sim \mathcal{D}, \mathbf{a} \sim \mu(\mathbf{a} \mid \mathbf{s})}[Q(\mathbf{s}, \mathbf{a})]+\frac{1}{2} \mathbb{E}_{\mathbf{s}, \mathbf{a} \sim \mathcal{D}}\left[\left(Q(\mathbf{s}, \mathbf{a})-\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})\right)^{2}\right] \tag{1}\label{1}$$</span></p>
  <p>My question is: what exactly is the derivative of equation (1)? And how does that result in equation (11)?</p>
  <p>The <span class="math-container">$\hat{B}^\pi$</span> is the empirical Bellman operator and is defined as <span class="math-container">$\hat{B}^\pi \hat{Q}^k (s,a) = r + \gamma \sum_{s'} \hat{T}(s' \mid s,a) \mathbb{E}_{a'\sim \pi(a' \mid s')}\hat{Q}_k(s', a')$</span>. Since in offline reinforcement learning, the dataset <span class="math-container">$\mathcal{D}$</span> typically does not contain all possible transitions <span class="math-container">$(s, a, s')$</span>, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-15T13:10:19.753Z
Comments.Text: Can you actually format the equations as text, rather than an image? That would make it a lot more accessible.
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-15T13:12:43.187Z
Comments.Text: Yes, please, next time, rather than providing the screenshots, for future reproducibility and easy accessibility, copy and quote the relevant parts of text from the paper, at least, as I've just done to your post. To get the equations in latex automatically, you can use a tool like "Mathpix Snipping Tool" (which is what I used).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-15T13:31:44.930Z
Comments.Text: I apologize, I will try to copy the relevant parts. And thanks for recommending mathpix snipping tools!
Comments.UserDisplayName: MoneyBall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-17T15:02:50.500Z
Comments.Text: Please can I confirm what happens to the expectations when the derivative is take w.r.t Q?
Comments.UserDisplayName: JoshML
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-27T21:52:40.423Z
Comments.Text: why can it ignore the expectation.
Comments.UserDisplayName: Wenze Chen
Tags: <reinforcement-learning><q-learning><papers><proofs>
Title: What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?
UserDisplayName: MoneyBall
ThreadText: # What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?
  
  (Asked by: username_0 on 2021-09-15)
  
  I am looking at the paper [Conservative Q-Learning for Offline Reinforcement Learning](https://arxiv.org/abs/2006.04779), but I'm not sure how they proved theorem 3.1.
  
  Here is a screenshot of theorem 3.1.
  
  [![enter image description here](https://i.sstatic.net/xBfaL.png)](https://i.sstatic.net/xBfaL.png)
  
  In the proof of theorem 3.1
  
  [![enter image description here](https://i.sstatic.net/GLIRy.png)](https://i.sstatic.net/GLIRy.png)
  
  they say
  
  > By setting the derivative of Equation 1 to 0, we obtain the following expression
  >
  > ...
  >
  > $$\forall \mathbf{s}, \mathbf{a} \in \mathcal{D}, k, \quad \hat{Q}^{k+1}(\mathbf{s}, \mathbf{a})=\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})-\alpha \frac{\mu(\mathbf{a} \mid \mathbf{s})}{\hat{\pi}\_{\beta}(\mathbf{a} \mid \mathbf{s})} \tag{11}\label{11}$$
  
  Here's equation 1 from the paper.
  
  $$\hat{Q}^{k+1} \leftarrow \arg \min \_{Q} \alpha \mathbb{E}\_{\mathbf{s} \sim \mathcal{D}, \mathbf{a} \sim \mu(\mathbf{a} \mid \mathbf{s})}[Q(\mathbf{s}, \mathbf{a})]+\frac{1}{2} \mathbb{E}\_{\mathbf{s}, \mathbf{a} \sim \mathcal{D}}\left[\left(Q(\mathbf{s}, \mathbf{a})-\hat{\mathcal{B}}^{\pi} \hat{Q}^{k}(\mathbf{s}, \mathbf{a})\right)^{2}\right] \tag{1}\label{1}$$
  
  My question is: what exactly is the derivative of equation (1)? And how does that result in equation (11)?
  
  The $\hat{B}^\pi$ is the empirical Bellman operator and is defined as $\hat{B}^\pi \hat{Q}^k (s,a) = r + \gamma \sum\_{s'} \hat{T}(s' \mid s,a) \mathbb{E}\_{a'\sim \pi(a' \mid s')}\hat{Q}\_k(s', a')$. Since in offline reinforcement learning, the dataset $\mathcal{D}$ typically does not contain all possible transitions $(s, a, s')$, the policy evaluation step actually uses an empirical Bellman operator that only backs up a single sample.
  
  **Tags:** reinforcement-learning, q-learning, papers, proofs
  > **Comments:**
  > **username_1**: Can you actually format the equations as text, rather than an image? That would make it a lot more accessible.  
  > **username_2**: Yes, please, next time, rather than providing the screenshots, for future reproducibility and easy accessibility, copy and quote the relevant parts of text from the paper, at least, as I've just done to your post. To get the equations in latex automatically, you can use a tool like "Mathpix Snipping Tool" (which is what I used).  
  > **username_0**: I apologize, I will try to copy the relevant parts. And thanks for recommending mathpix snipping tools!  
  > **username_3**: Please can I confirm what happens to the expectations when the derivative is take w.r.t Q?  
  > **username_4**: why can it ignore the expectation.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_5 on 2021-12-15. Score: 0
  
  first we need to transform the distribution of the first term:
  
  $\mathrm{argmin} \space \alpha \mathrm{E}\_{s\sim D, a \sim D}[\frac{\mu}{\hat\pi\_{\beta}}Q(s,a)] + \frac{1}{2}\mathrm{E}\_{s,a \sim D}(Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a))^2 $
  
  then its derivate respect to Q is:
  
  $\alpha\frac{\mu}{\hat \pi ^{\beta}} + Q(s,a)-\hat\beta^{\pi}\hat Q^k(s,a) = 0$
  
  then we have
  
  $ Q(s,a) =\hat\beta^{\pi}\hat Q^k(s,a) -\alpha\frac{\mu}{\hat \pi ^{\beta}}$
  
  > **Comments:**
  > **username_0**: how did you find the derivative with respect to Q?  
  
  ---
  
  