Answers.Body: <p>Softmax policy <span class="math-container">$\pi_\theta(s,a)$</span> is defined as <span class="math-container">$\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$</span>, where the summation is over the action space.<br>
  Taking log, this becomes
  <span class="math-container">$$
  \log \pi_\theta(s,a) = log(e^{\phi(s,a) ^T \theta}) - log({\Sigma e^{\phi(s,a) ^T \theta }}) \\
  = \phi(s,a) ^T \theta - log({\Sigma e^{\phi(s,a)^T \theta }})
  $$</span></p>
  
  <p>Taking derivative wrt <span class="math-container">$\theta$</span>, this becomes
  <span class="math-container">$$
  \nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \nabla_\theta log({\Sigma e^{\phi(s,a) ^T \theta }})
  $$</span></p>
  
  <p>We can rewrite <span class="math-container">$\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }})$</span> as follows.
  <span class="math-container">$$
  \nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }}) 
  = \frac{\nabla_\theta \Sigma e^{\phi(s,a)^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}} 
  = \frac{\Sigma \phi(s,a) e^{\phi(s,a) ^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}}
  = \Sigma \phi(s,a) \pi_\theta(s,a)
  $$</span> </p>
  
  <p>The final equation then becomes
  <span class="math-container">$$
  \nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \Sigma \phi(s,a) \pi_\theta(s,a)
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T19:36:40.797Z
Answers.Text: I'd recommend writing each of the $e^{\phi(s, a) \theta}$ things as $\exp(\phi(s, a)^{\top} \theta)$. Not having to write it in the smaller superscript makes it a bit more readable, especially for the in-text equations. And I like making the transpose a bit more explicitly visible too personally :) That last part is assuming we're dealing with feature and weight vectors of course, not just scalars
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T19:31:41.313Z
Answers.LastActivityDate: 2020-05-21T20:08:46.850Z
Answers.LastEditDate: 2020-05-21T20:08:46.850Z
Answers.UserDisplayName: pecey
Body: <p>The gradient of the softmax eligibility trace is given by the following:</p>
  
  <p><span class="math-container">\begin{align}
  \nabla_{\theta} \log(\pi_{\theta}(a|s)) &amp;= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\
  &amp;= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
  \end{align}</span></p>
  
  <p>How is this equation derived?</p>
  
  <p>The following relation is true:</p>
  
  <p><span class="math-container">\begin{align}
  \nabla_{\theta} \log(\pi_{\theta}(a|s)) &amp;= \frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)} \tag{1}\label{1}
  \end{align}</span></p>
  
  <p>Thus, the following relation must also be true:
  <span class="math-container">\begin{align}
  \frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)}  &amp;=\phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
  \end{align}</span></p>
  
  <p>Mathematically, why would this be the case? Probably, you just need to answer my question above because \ref{1} is true and it's just <a href="https://ai.stackexchange.com/a/20665/2444">the rule to differentiate a logarithm</a>.</p>
  
Tags: <reinforcement-learning><policy-gradients><eligibility-traces>
Title: How do I derive the gradient with respect to the parameters of the softmax policy?
UserDisplayName: Stephane Hatgis-Kessell
ThreadText: # How do I derive the gradient with respect to the parameters of the softmax policy?
  
  (Asked by: username_0 on 2020-05-19)
  
  The gradient of the softmax eligibility trace is given by the following:
  
  \begin{align}
  \nabla\_{\theta} \log(\pi\_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\
  &= \phi(s,a) - \sum\_{a'} \pi(a'|s) \phi(s,a')
  \end{align}
  
  How is this equation derived?
  
  The following relation is true:
  
  \begin{align}
  \nabla\_{\theta} \log(\pi\_{\theta}(a|s)) &= \frac{\nabla\_{\theta} \pi\_{\theta}(a|s)}{\pi\_{\theta}(a|s)} \tag{1}\label{1}
  \end{align}
  
  Thus, the following relation must also be true:
  \begin{align}
  \frac{\nabla\_{\theta} \pi\_{\theta}(a|s)}{\pi\_{\theta}(a|s)} &=\phi(s,a) - \sum\_{a'} \pi(a'|s) \phi(s,a')
  \end{align}
  
  Mathematically, why would this be the case? Probably, you just need to answer my question above because \ref{1} is true and it's just [the rule to differentiate a logarithm](https://ai.stackexchange.com/a/20665/2444).
  
  **Tags:** reinforcement-learning, policy-gradients, eligibility-traces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-21. Score: 3 | Accepted answer
  
  Softmax policy $\pi\_\theta(s,a)$ is defined as $\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$, where the summation is over the action space.  
  Taking log, this becomes
  $$
  \log \pi\_\theta(s,a) = log(e^{\phi(s,a) ^T \theta}) - log({\Sigma e^{\phi(s,a) ^T \theta }}) \\
  = \phi(s,a) ^T \theta - log({\Sigma e^{\phi(s,a)^T \theta }})
  $$
  
  Taking derivative wrt $\theta$, this becomes
  $$
  \nabla\_\theta \log \pi\_\theta(s,a) = \phi(s,a) - \nabla\_\theta log({\Sigma e^{\phi(s,a) ^T \theta }})
  $$
  
  We can rewrite $\nabla\_\theta log({\Sigma e^{\phi(s,a)^T \theta }})$ as follows.
  $$
  \nabla\_\theta log({\Sigma e^{\phi(s,a)^T \theta }})
  = \frac{\nabla\_\theta \Sigma e^{\phi(s,a)^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}}
  = \frac{\Sigma \phi(s,a) e^{\phi(s,a) ^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}}
  = \Sigma \phi(s,a) \pi\_\theta(s,a)
  $$
  
  The final equation then becomes
  $$
  \nabla\_\theta \log \pi\_\theta(s,a) = \phi(s,a) - \Sigma \phi(s,a) \pi\_\theta(s,a)
  $$
  
  > **Comments:**
  > **username_2**: I'd recommend writing each of the $e^{\phi(s, a) \theta}$ things as $\exp(\phi(s, a)^{\top} \theta)$. Not having to write it in the smaller superscript makes it a bit more readable, especially for the in-text equations. And I like making the transpose a bit more explicitly visible too personally :) That last part is assuming we're dealing with feature and weight vectors of course, not just scalars  
  
  ---
  
  