Answers.Body: <p>The paper only briefly explains the quoted derivation is mainly because it's only standard application of the policy gradient theorem (PGT) for the proposed maximum entropy RL here where you can move the gradient operator inside the expectation statistic acting on the log policy distribution <span class="math-container">$q_{\theta}$</span>. The only difference with the standard PGT is that here the reward term at each time step is added to a <span class="math-container">$\theta$</span>-dependent Shannon entropy term, thus apparently you have to take additional gradient of the entropy term.</p>
  <p>Framed this way the innermost summation terms including <span class="math-container">$-1$</span> of the second step is just the expansion from the RHS of the first step based upon entropy definition <span class="math-container">$\mathcal{H}$</span> for the policy and taking derivative of <span class="math-container">$\mathcal{H}$</span> wrt <span class="math-container">$\theta$</span> as also explained by the authors immediately after your quoted equations, where please note they omitted parentheses. They also explained the last line follows by noting that the gradient estimator is invariant to additive state dependent constants at each time step <span class="math-container">$t'$</span>, and thus replacing <span class="math-container">$−1$</span> with a state-dependent baseline <span class="math-container">$b(s_{t′})$</span>, which is very similar to the standard REINFORCE with Baseline method for reducing variance during Monte Carlo sampling.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-19T21:32:23.680Z
Answers.LastActivityDate: 2024-07-19T22:26:20.860Z
Answers.LastEditDate: 2024-07-19T22:26:20.860Z
Answers.UserDisplayName: cinch
Body: <p>This comes from the paper, <a href="https://arxiv.org/pdf/1805.00909" rel="nofollow noreferrer">Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review</a>.</p>
  <p>I don't know the why the following derivation is true. The paper only briefly explains this.</p>
  <p>Can someone help with a detailed deducing process?</p>
  <p><a href="https://i.sstatic.net/mWmirPDs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mWmirPDs.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><papers><math>
Title: Why is this RL derivation right?
UserDisplayName: yeebo xie
ThreadText: # Why is this RL derivation right?
  
  (Asked by: username_0 on 2024-07-17)
  
  This comes from the paper, [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://arxiv.org/pdf/1805.00909).
  
  I don't know the why the following derivation is true. The paper only briefly explains this.
  
  Can someone help with a detailed deducing process?
  
  [![enter image description here](https://i.sstatic.net/mWmirPDs.png)](https://i.sstatic.net/mWmirPDs.png)
  
  **Tags:** reinforcement-learning, papers, math
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-19. Score: 1 | Accepted answer
  
  The paper only briefly explains the quoted derivation is mainly because it's only standard application of the policy gradient theorem (PGT) for the proposed maximum entropy RL here where you can move the gradient operator inside the expectation statistic acting on the log policy distribution $q\_{\theta}$. The only difference with the standard PGT is that here the reward term at each time step is added to a $\theta$-dependent Shannon entropy term, thus apparently you have to take additional gradient of the entropy term.
  
  Framed this way the innermost summation terms including $-1$ of the second step is just the expansion from the RHS of the first step based upon entropy definition $\mathcal{H}$ for the policy and taking derivative of $\mathcal{H}$ wrt $\theta$ as also explained by the authors immediately after your quoted equations, where please note they omitted parentheses. They also explained the last line follows by noting that the gradient estimator is invariant to additive state dependent constants at each time step $t'$, and thus replacing $−1$ with a state-dependent baseline $b(s\_{t′})$, which is very similar to the standard REINFORCE with Baseline method for reducing variance during Monte Carlo sampling.
  
  
  ---
  
  