Body: <p><a href="http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching_files/pg.pdf" rel="nofollow noreferrer">At slide 17</a> of the David Silver's series, the soft-max policy is defined as follows</p>
  
  <p><span class="math-container">$$
  \pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
  $$</span></p>
  
  <p>that is, the probability of an action <span class="math-container">$a$</span> (in state <span class="math-container">$s$</span>) is proportional to the "exponentiated weight".</p>
  
  <p>The score function is then defined as follows</p>
  
  <p><span class="math-container">$$
  \nabla_\theta \log \pi_\theta (s, a) = \phi(s, a) - 
  \mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)]
  $$</span></p>
  
  <p>Where does the expectation term <span class="math-container">$\mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)]$</span> come from?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-15T19:06:56.363Z
Comments.Text: see this answer over on the math.stackexchange.com site: https://math.stackexchange.com/a/2340848/414001
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-17T16:15:24.117Z
Comments.Text: I'm not 100% sure what you are asking but I'll try to answer it anyway.
  The expectation term comes from the derivative of the softmax function. This video from Geoffrey Hinton should help: https://www.youtube.com/watch?v=mlaLLQofmR8
Comments.UserDisplayName: Jaden Travnik
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-17T17:58:49.103Z
Comments.Text: The confusion here is that, when we take log of pi, then it will log(sum x_i) in dem, where x_i is that expression. Now my doubt is here this term log(sum_i  x_i) is replace by sum_i (log x_i), how? Is this inequality valid?
Comments.UserDisplayName: Ankish Bansal
Tags: <reinforcement-learning><math><policies><expectation>
Title: Where does the expectation term in the derivative of the soft-max policy come from?
UserDisplayName: Ankish Bansal
ThreadText: # Where does the expectation term in the derivative of the soft-max policy come from?
  
  (Asked by: username_0 on 2019-01-17)
  
  [At slide 17](http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching_files/pg.pdf) of the David Silver's series, the soft-max policy is defined as follows
  
  $$
  \pi\_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
  $$
  
  that is, the probability of an action $a$ (in state $s$) is proportional to the "exponentiated weight".
  
  The score function is then defined as follows
  
  $$
  \nabla\_\theta \log \pi\_\theta (s, a) = \phi(s, a) -
  \mathbb{E}\_{\pi\_{\theta}}[\phi(s, \cdot)]
  $$
  
  Where does the expectation term $\mathbb{E}\_{\pi\_{\theta}}[\phi(s, \cdot)]$ come from?
  
  **Tags:** reinforcement-learning, math, policies, expectation
  
  > **Comments:**
  > **username_2**: I'm not 100% sure what you are asking but I'll try to answer it anyway. The expectation term comes from the derivative of the softmax function. This video from Geoffrey Hinton should help: https://www.youtube.com/watch?v=mlaLLQofmR8  
  > **username_0**: The confusion here is that, when we take log of pi, then it will log(sum x_i) in dem, where x_i is that expression. Now my doubt is here this term log(sum_i  x_i) is replace by sum_i (log x_i), how? Is this inequality valid?  
  > **username_1**: see this answer over on the math.stackexchange.com site: https://math.stackexchange.com/a/2340848/414001  
  
  ---
  
  ## 0 answers
  
  