Answers.Body: <p>Chiming in because I had the same question and stumbled across your post. It seems like the general version of your question still has not been answered.</p>
  <p>In general, a well-formed gradient update rule is all you need to be able to train the network. We are thinking of converting to a &quot;loss function&quot; because that is the typical flow in the structure that pytorch provides for training networks via their autograd framework.</p>
  <p>But the pytorch loss function is really just meant to be the last in a list of nested operations over which we are going to compute gradients. So if you already know your gradient update rule, the easiest way to write it down as a pytorch &quot;loss function&quot; is as the integral of your gradient update rule with respect to the network's output -- then pytorch will compute your gradient update rule for you automatically.</p>
  <p>So in your example, the loss function would be:</p>
  <p><span class="math-container">$-\alpha^tG \ln\pi(A_t | S_t, \theta_t)$</span></p>
  <p>as already noted by @Brale and others. The minus sign is there because pytorch optimizers will typically carry out minimization, but your gradient update rule is intended to maximize.</p>
  <p>Hopefully this explanation is helpful to others that are still learning pytorch! Also, I would highly recommend that anyone just getting started with pytorch read their latest autograd tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py" rel="nofollow noreferrer">here</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T01:51:16.357Z
Answers.LastActivityDate: 2021-04-20T01:51:16.357Z
Answers.UserDisplayName: killian95
Answers.Body: <p>You can find an implementation of the REINFORCE algorithm (as defined in your question) in PyTorch at the following URL: <a href="https://github.com/JamesChuanggg/pytorch-REINFORCE/" rel="nofollow noreferrer">https://github.com/JamesChuanggg/pytorch-REINFORCE/</a>. First of all, I would like to note that a policy can be represented or implemented as a neural network, where the input is the state (you are currently in) and the output is a "probability distribution over the actions you can take from that state received as input". </p>
  
  <p>In the Python module <a href="https://github.com/JamesChuanggg/pytorch-REINFORCE/blob/master/reinforce_discrete.py" rel="nofollow noreferrer">https://github.com/JamesChuanggg/pytorch-REINFORCE/blob/master/reinforce_discrete.py</a>, the policy is defined as a neural network with 2 linear layers, where the first linear layer is followed by a ReLU activation function, whereas the second is followed by a soft-max. In that same Python module, the author also defines another class called <code>REINFORCE</code>, which creates a <code>Policy</code> object (in the <code>__init__</code> method) and defines it as property of that class. The class <code>REINFORCE</code> also defines two methods <code>select_action</code> and <code>update_parameters</code>. These two methods are called from the <a href="https://github.com/JamesChuanggg/pytorch-REINFORCE/blob/master/main.py" rel="nofollow noreferrer"><code>main.py</code></a> module, where the main loop of the <code>REINFORCE</code> algorithm is implemented. In that same main loop, the author declares lists <code>entropies</code>, <code>log_probs</code> and <code>rewards</code>. Note that these lists are re-initialized at ever episode. A "log_prob" and an "entropy" is returned from the <code>select_action</code> method, whereas a "reward" is returned from the environment after having executed one environment step. The environment is provided by the OpenAI's Gym library. The lists <code>entropies</code>, <code>log_probs</code> and <code>rewards</code> are then used to update the parameters, i.e. they are used by the method <code>update_parameters</code> defined in the class <code>REINFORCE</code>. </p>
  
  <p>Let's see better now what these methods, <code>select_action</code> and <code>update_parameters</code>, actually do. </p>
  
  <p><code>select_action</code> first calls the <code>forward</code> method of the class <code>Policy</code>, which returns the output of the forward pass of the NN (i.e. the output of the soft-max layer), so it returns the probabilities of selecting each of the available actions (from the state given as input). It then selects the probability associated with the first action (I guess, it picks the probabilities associated with the action with the highest probabilities), denoted by <code>prob</code> (in the source code). Essentially, what I've described so far regarding this <code>select_action</code> method is the computation of <span class="math-container">$\pi(A_t \mid S_t, \theta)$</span> (as shown in the pseudocode of your question). Afterwards, in the same method <code>select_action</code>, the author also computes the log of that probability I've just mentioned above (i.e. the one associated with the  action with the highest probability, i.e. the log of <code>prob</code>), denoted by <code>log_prob</code>. In that same method, the entropy (as defined in <a href="https://datascience.stackexchange.com/a/20301/10640">this answer</a>) is calculated. In reality, the author calculates the entropy using only one distribution (instead of two): more specifically, the entropy is calculated as follows <code>entropy = -(probs*probs.log()).sum()</code>. In fact, the entropy loss function usually requires the ground-truth labels (as explained in the answer I linked you to above), but, in this case, we do not have ground-truth labels (given that we are performing RL and not supervised learning). Nonetheless, I can't really tell you why the entropy is calculated like this, in this case. Finally, the method <code>select_action</code> then return <code>action[0], log_prob, entropy</code>.</p>
  
  <p>First of all, I would like to note that the method <code>update_parameters</code> is called only at the end of each episode (in the <code>main.py</code> module). In that same method, a variable called <code>loss</code> is first initialized to zero. In that method, we then iterate the list of rewards for the current episode. Inside that loop of the <code>update_parameters</code> method, the return, <code>R</code> is calculated. <code>R</code> is also multiplied by <span class="math-container">$\gamma$</span>. On each time step, the loss is then calculated as follows</p>
  
  <pre><code>loss = loss - (log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()
  </code></pre>
  
  <p>The <code>loss</code> is calculated by subtracting the previous <code>loss</code> with </p>
  
  <pre><code>(log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()
  </code></pre>
  
  <p>where <code>log_probs</code> are the log probabilities calculated in the <code>select_action</code> method. <code>log_probs</code> is the part <span class="math-container">$\log \pi(A_t \mid S_t, \theta)$</span> of the update rule of your pseudocode. <code>log_probs</code> are then multiplied by the return <code>R</code>. We then sum the result of this multiplication over all elements of the vector. We then subtract this just obtained result by the entropies multiplied by 0.0001. I can't really tell you why the author decided to implement the loss in this way. I would need to think about it a little more.</p>
  
  <p>The following article may also be useful: <a href="https://pytorch.org/docs/stable/distributions.html" rel="nofollow noreferrer">https://pytorch.org/docs/stable/distributions.html</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-14T21:18:45.273Z
Answers.Text: @Gulzar If you have figured out more details behind my explanations and the linked source code, feel free to add them as an answer to your own question!
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-14T22:59:17.647Z
Answers.Text: Sorry about that, i had to put that project aside for too many weeks and. I hope i will get back to it asap
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-21T01:54:58.317Z
Answers.Text: @nbro the entropy function you describe is the information theory entropy function found [here](https://en.m.wikipedia.org/wiki/Entropy_(information_theory)). The entropy function you refer to with two distributions is the Cross-entropy function, which essentially compares how similar two distributions are. The entropy function used here is the average rate at which information is produced by a stochastic source of data.
Answers.UserDisplayName: Hanzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-21T03:52:58.427Z
Answers.Text: The only part of the loss function I can’t discern is why the author subtracts (0.0001*entropies[i]) at each step of calculating the loss.
Answers.UserDisplayName: Hanzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-14T16:51:03.803Z
Answers.Text: Thanks for all the details! However, my question was mostly theoretical. In most papers i've seen, the update rules are formulated like the pseudocode equations I linked in the question. I wanted to know how IN THE GENERAL CASE I would convert such equations to a loss function. Pseudocode would be enough for this purpose.
  I'm sorry, but I still can't figure out the equation which `loss=...` came from.
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T18:09:22.193Z
Answers.LastActivityDate: 2019-02-12T18:09:22.193Z
Answers.UserDisplayName: nbro
Body: <p>From <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">the reinforcement learning book section 13.3</a>:</p>
  
  <p><a href="https://i.sstatic.net/Ex4a5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ex4a5.png" alt="enter image description here"></a></p>
  
  <p>Using pytorch, I need to calculate a loss, and then the gradient is calculated internally.</p>
  
  <p>How to obtain the loss from equations which are stated in the form of an iterative update with respect to the gradient?</p>
  
  <p>In this case:</p>
  
  <p><span class="math-container">$\theta \leftarrow \theta + \alpha\gamma^tG\nabla_{\theta}ln\pi(A_t|S_t,\theta)$</span></p>
  
  <p>What would be the loss?</p>
  
  <p>And in general, what would be the loss if the update rule were</p>
  
  <p><span class="math-container">$\theta \leftarrow \theta + \alpha C\nabla_{\theta}g(x|\theta)$</span></p>
  
  <p>for some general (derivable) function <span class="math-container">$g$</span> parameterized by theta?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-14T20:58:41.130Z
Comments.Text: The minus loss will be expectation of accumulated reward or accumulated discounted reward over parametrized policy. It's explained usually in "log derivative trick" chapters, for example here:  http://www.1-4-5.net/~dmm/ml/log_derivative_trick.pdf
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-12T13:08:00.933Z
Comments.Text: Why do you need to calculate any loss here? You need to compute the derivate with respect to $\theta$ of $\ln \pi (A_t \mid S_t, \theta)$. I suppose that $\pi (A_t \mid S_t, \theta)$ will be represented as a neural network. You're not performing supervised learning.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-12T13:17:32.993Z
Comments.Text: If I were to implement this with pytorch, I would need to calculate some kind of loss in order to update the policy $\pi$. I simply don't quite understand how to transform the iterative notation to a loss notation (which would then be fed into the optimizer)
Comments.UserDisplayName: Gulzar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-12T15:46:07.320Z
Comments.Text: Have you had a look at the examples here: https://github.com/pytorch/examples/tree/master/reinforcement_learning?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-12T15:49:44.180Z
Comments.Text: @nbro this is my only question here in which I haven't linked to one of the code examples there :). My question is more general, because I want to go on to the next algorithms in the book, and then onwards for example algorithm 13.5, for which there is no code in the linked pytorch example.
Comments.UserDisplayName: Gulzar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-12T16:51:26.313Z
Comments.Text: your "loss" would be $-ln(\pi(a\mid s, \theta))G\gamma^t$, I somewhat answered your question [here](https://ai.stackexchange.com/questions/10336/inconsistent-formulas-for-loss-calculation-in-openais-actor-critic/10342#10342). In this case $A$ that i wrote there would be $G\gamma^t$
Comments.UserDisplayName: Brale
Tags: <machine-learning><reinforcement-learning><gradient-descent><objective-functions>
Title: How to obtain a formula for loss, when given an iterative update rule in gradient descent?
UserDisplayName: Gulzar
ThreadText: # How to obtain a formula for loss, when given an iterative update rule in gradient descent?
  
  (Asked by: username_0 on 2019-02-12)
  
  From [the reinforcement learning book section 13.3](http://incompleteideas.net/book/bookdraft2017nov5.pdf):
  
  [![enter image description here](https://i.sstatic.net/Ex4a5.png)](https://i.sstatic.net/Ex4a5.png)
  
  Using pytorch, I need to calculate a loss, and then the gradient is calculated internally.
  
  How to obtain the loss from equations which are stated in the form of an iterative update with respect to the gradient?
  
  In this case:
  
  $\theta \leftarrow \theta + \alpha\gamma^tG\nabla\_{\theta}ln\pi(A\_t|S\_t,\theta)$
  
  What would be the loss?
  
  And in general, what would be the loss if the update rule were
  
  $\theta \leftarrow \theta + \alpha C\nabla\_{\theta}g(x|\theta)$
  
  for some general (derivable) function $g$ parameterized by theta?
  
  **Tags:** machine-learning, reinforcement-learning, gradient-descent, objective-functions
  > **Comments:**
  > **username_2**: Why do you need to calculate any loss here? You need to compute the derivate with respect to $\theta$ of $\ln \pi (A_t \mid S_t, \theta)$. I suppose that $\pi (A_t \mid S_t, \theta)$ will be represented as a neural network. You're not performing supervised learning.  
  > **username_0**: If I were to implement this with pytorch, I would need to calculate some kind of loss in order to update the policy $\pi$. I simply don't quite understand how to transform the iterative notation to a loss notation (which would then be fed into the optimizer)  
  > **username_2**: Have you had a look at the examples here: https://github.com/pytorch/examples/tree/master/reinforcement_learning?  
  > **username_0**: @nbro this is my only question here in which I haven't linked to one of the code examples there :). My question is more general, because I want to go on to the next algorithms in the book, and then onwards for example algorithm 13.5, for which there is no code in the linked pytorch example.  
  > **username_3**: your "loss" would be $-ln(\pi(a\mid s, \theta))G\gamma^t$, I somewhat answered your question [here](https://ai.stackexchange.com/questions/10336/inconsistent-formulas-for-loss-calculation-in-openais-actor-critic/10342#10342). In this case $A$ that i wrote there would be $G\gamma^t$  
  > **username_1**: The minus loss will be expectation of accumulated reward or accumulated discounted reward over parametrized policy. It's explained usually in "log derivative trick" chapters, for example here:  http://www.1-4-5.net/~dmm/ml/log_derivative_trick.pdf  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-02-12. Score: 3 | Accepted answer
  
  You can find an implementation of the REINFORCE algorithm (as defined in your question) in PyTorch at the following URL: https://github.com/JamesChuanggg/pytorch-REINFORCE/. First of all, I would like to note that a policy can be represented or implemented as a neural network, where the input is the state (you are currently in) and the output is a "probability distribution over the actions you can take from that state received as input". 
  In the Python module https://github.com/JamesChuanggg/pytorch-REINFORCE/blob/master/reinforce_discrete.py, the policy is defined as a neural network with 2 linear layers, where the first linear layer is followed by a ReLU activation function, whereas the second is followed by a soft-max. In that same Python module, the author also defines another class called REINFORCE, which creates a Policy object (in the __init__ method) and defines it as property of that class. The class REINFORCE also defines two methods select_action and update_parameters. These two methods are called from the main.py module, where the main loop of the REINFORCE algorithm is implemented. In that same main loop, the author declares lists entropies, log_probs and rewards. Note that these lists are re-initialized at ever episode. A "log_prob" and an "entropy" is returned from the select_action method, whereas a "reward" is returned from the environment after having executed one environment step. The environment is provided by the OpenAI's Gym library. The lists entropies, log_probs and rewards are then used to update the parameters, i.e. they are used by the method update_parameters defined in the class REINFORCE. 
  Let's see better now what these methods, select_action and update_parameters, actually do. 
  select_action first calls the forward method of the class Policy, which returns the output of the forward pass of the NN (i.e. the output of the soft-max layer), so it returns the probabilities of selecting each of the available actions (from the state given as input). It then selects the probability associated with the first action (I guess, it picks the probabilities associated with the action with the highest probabilities), denoted by prob (in the source code). Essentially, what I've described so far regarding this select_action method is the computation of $\pi(A_t \mid S_t, \theta)$ (as shown in the pseudocode of your question). Afterwards, in the same method select_action, the author also computes the log of that probability I've just mentioned above (i.e. the one associated with the  action with the highest probability, i.e. the log of prob), denoted by log_prob. In that same method, the entropy (as defined in this answer) is calculated. In reality, the author calculates the entropy using only one distribution (instead of two): more specifically, the entropy is calculated as follows entropy = -(probs*probs.log()).sum(). In fact, the entropy loss function usually requires the ground-truth labels (as explained in the answer I linked you to above), but, in this case, we do not have ground-truth labels (given that we are performing RL and not supervised learning). Nonetheless, I can't really tell you why the entropy is calculated like this, in this case. Finally, the method select_action then return action[0], log_prob, entropy.
  First of all, I would like to note that the method update_parameters is called only at the end of each episode (in the main.py module). In that same method, a variable called loss is first initialized to zero. In that method, we then iterate the list of rewards for the current episode. Inside that loop of the update_parameters method, the return, R is calculated. R is also multiplied by $\gamma$. On each time step, the loss is then calculated as follows
  loss = loss - (log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()
  
  The loss is calculated by subtracting the previous loss with 
  (log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()
  
  where log_probs are the log probabilities calculated in the select_action method. log_probs is the part $\log \pi(A_t \mid S_t, \theta)$ of the update rule of your pseudocode. log_probs are then multiplied by the return R. We then sum the result of this multiplication over all elements of the vector. We then subtract this just obtained result by the entropies multiplied by 0.0001. I can't really tell you why the author decided to implement the loss in this way. I would need to think about it a little more.
  The following article may also be useful: https://pytorch.org/docs/stable/distributions.html.
  
  > **Comments:**
  > **username_0**: Thanks for all the details! However, my question was mostly theoretical. In most papers i've seen, the update rules are formulated like the pseudocode equations I linked in the question. I wanted to know how IN THE GENERAL CASE I would convert such equations to a loss function. Pseudocode would be enough for this purpose. I'm sorry, but I still can't figure out the equation which `loss=...` came from.  
  > **username_2**: @Gulzar If you have figured out more details behind my explanations and the linked source code, feel free to add them as an answer to your own question!  
  > **username_0**: Sorry about that, i had to put that project aside for too many weeks and. I hope i will get back to it asap  
  > **username_5**: @nbro the entropy function you describe is the information theory entropy function found [here](https://en.m.wikipedia.org/wiki/Entropy_(information_theory)). The entropy function you refer to with two distributions is the Cross-entropy function, which essentially compares how similar two distributions are. The entropy function used here is the average rate at which information is produced by a stochastic source of data.  
  > **username_5**: The only part of the loss function I can’t discern is why the author subtracts (0.0001*entropies[i]) at each step of calculating the loss.  
  
  ---
  
  ### Answer by username_4 on 2021-04-20. Score: 0
  
  Chiming in because I had the same question and stumbled across your post. It seems like the general version of your question still has not been answered.
  
  In general, a well-formed gradient update rule is all you need to be able to train the network. We are thinking of converting to a "loss function" because that is the typical flow in the structure that pytorch provides for training networks via their autograd framework.
  
  But the pytorch loss function is really just meant to be the last in a list of nested operations over which we are going to compute gradients. So if you already know your gradient update rule, the easiest way to write it down as a pytorch "loss function" is as the integral of your gradient update rule with respect to the network's output -- then pytorch will compute your gradient update rule for you automatically.
  
  So in your example, the loss function would be:
  
  $-\alpha^tG \ln\pi(A\_t | S\_t, \theta\_t)$
  
  as already noted by @Brale and others. The minus sign is there because pytorch optimizers will typically carry out minimization, but your gradient update rule is intended to maximize.
  
  Hopefully this explanation is helpful to others that are still learning pytorch! Also, I would highly recommend that anyone just getting started with pytorch read their latest autograd tutorial [here](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py)
  
  
  ---
  
  