Answers.Body: <p>In reinforcement learning, you can distinguish algorithms based on the functions they use to ultimately find the policy (which is the goal in RL anyway!).</p>
  
  <ul>
  <li>algorithms that attempt to find an optimal value function (an example is Q-learning, which attempts to find a state-action value function), then derive the policy from the value function</li>
  <li>algorithms that directly attempt to find a policy (e.g. REINFORCE and other so-called "policy gradients" algorithms)</li>
  <li>algorithms that use a value function to guide the search for an optimal policy (i.e. actor-critic methods)</li>
  </ul>
  
  <p>More specifically, in actor-critic methods, you have a policy <span class="math-container">$\pi$</span> (known as the "actor") and a value function <span class="math-container">$v$</span> (known as the "critic"). Hence the name "actor-critic". The idea is that you will use this critic <span class="math-container">$v$</span> to "criticize" the policy (or actor) <span class="math-container">$\pi$</span>, i.e. to guide the search for a good policy. This article <a href="http://incompleteideas.net/book/first/ebook/node66.html" rel="nofollow noreferrer">6.6 Actor-Critic Methods</a> (from Sutton and Barto's book) explains the concept quite well.</p>
  
  <p>In your specific example, the policy and value function are assumed to be differentiable (otherwise, you wouldn't be able to compute the derivatives anyway!). They are typically neural networks. <span class="math-container">$\theta_\pi$</span> are the parameters of the neural network that represent the policy (i.e. a neural network that receives as input a state and produces as output an action or probability distribution over actions). Similarly, <span class="math-container">$\theta_v$</span> are the parameters of the neural network that represent the critic. Then <span class="math-container">$\partial \theta_\pi$</span> and <span class="math-container">$\partial \theta_\pi $</span> will represent an accumulation of the gradients with respect to the actor and critic,  respectively. (The accumulation is probably over the steps from <span class="math-container">$i=t-1$</span> to <span class="math-container">$i = t_{\text{start}}$</span>, but I can't say more because I never implemented actor-critic methods). This should give you an idea, though!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-24T19:58:11.827Z
Answers.LastActivityDate: 2020-04-24T20:11:18.360Z
Answers.LastEditDate: 2020-04-24T20:11:18.360Z
Answers.UserDisplayName: nbro
Body: <p>One of the steps in the actor-critic algorithm is <span class="math-container">$$\partial \theta_{\pi} \gets \partial \theta_{\pi} + \nabla_{\theta}\log\pi_{\theta} (a_i | s_i) (R - V_{\theta}(s_i))$$</span></p>
  
  <p>For me, <span class="math-container">$\theta$</span> are just the weights. Can you explain to me what mean <span class="math-container">$\partial \theta_{\pi}$</span>?</p>
  
  <p>The whole algorithm comes from Maxim Lepan's book <em>Deep Reinforcement Learning Hands-on</em> page 269.</p>
  
  <p>Here is a picture of the algorithm : </p>
  
  <p><a href="https://i.sstatic.net/TbZwe.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TbZwe.jpg" alt="enter image description here"></a></p>
  
Tags: <reinforcement-learning><deep-rl><pytorch><actor-critic-methods><notation>
Title: What does the notation $\partial \theta_{\pi}$ mean in this actor-critic update rule?
UserDisplayName: jgauth
ThreadText: # What does the notation $\partial \theta_{\pi}$ mean in this actor-critic update rule?
  
  (Asked by: username_0 on 2020-04-24)
  
  One of the steps in the actor-critic algorithm is $$\partial \theta\_{\pi} \gets \partial \theta\_{\pi} + \nabla\_{\theta}\log\pi\_{\theta} (a\_i | s\_i) (R - V\_{\theta}(s\_i))$$
  
  For me, $\theta$ are just the weights. Can you explain to me what mean $\partial \theta\_{\pi}$?
  
  The whole algorithm comes from Maxim Lepan's book *Deep Reinforcement Learning Hands-on* page 269.
  
  Here is a picture of the algorithm :
  
  [![enter image description here](https://i.sstatic.net/TbZwe.jpg)](https://i.sstatic.net/TbZwe.jpg)
  
  **Tags:** reinforcement-learning, deep-rl, pytorch, actor-critic-methods, notation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-24. Score: 1 | Accepted answer
  
  In reinforcement learning, you can distinguish algorithms based on the functions they use to ultimately find the policy (which is the goal in RL anyway!).
  
  * algorithms that attempt to find an optimal value function (an example is Q-learning, which attempts to find a state-action value function), then derive the policy from the value function
  * algorithms that directly attempt to find a policy (e.g. REINFORCE and other so-called "policy gradients" algorithms)
  * algorithms that use a value function to guide the search for an optimal policy (i.e. actor-critic methods)
  
  More specifically, in actor-critic methods, you have a policy $\pi$ (known as the "actor") and a value function $v$ (known as the "critic"). Hence the name "actor-critic". The idea is that you will use this critic $v$ to "criticize" the policy (or actor) $\pi$, i.e. to guide the search for a good policy. This article [6.6 Actor-Critic Methods](http://incompleteideas.net/book/first/ebook/node66.html) (from Sutton and Barto's book) explains the concept quite well.
  
  In your specific example, the policy and value function are assumed to be differentiable (otherwise, you wouldn't be able to compute the derivatives anyway!). They are typically neural networks. $\theta\_\pi$ are the parameters of the neural network that represent the policy (i.e. a neural network that receives as input a state and produces as output an action or probability distribution over actions). Similarly, $\theta\_v$ are the parameters of the neural network that represent the critic. Then $\partial \theta\_\pi$ and $\partial \theta\_\pi $ will represent an accumulation of the gradients with respect to the actor and critic, respectively. (The accumulation is probably over the steps from $i=t-1$ to $i = t\_{\text{start}}$, but I can't say more because I never implemented actor-critic methods). This should give you an idea, though!
  
  
  ---
  
  