Answers.Body: <p>This update rule can still be applied in the continuous domain.</p>
  <p>As pointed out in the comments, suppose we are parameterising our policy using a Gaussian distribution, where our neural networks take as input the state we are in and output the parameters of a Gaussian distribution, the mean and the standard deviation which we will denote as <span class="math-container">$\mu(s, \theta)$</span> and <span class="math-container">$\sigma(s, \theta)$</span> where <span class="math-container">$s$</span> shows the dependancy of the state and <span class="math-container">$\theta$</span> are the parameters of our network.</p>
  <p>I will assume a one-dimensional case for ease of notation but this can be extended to multi-variate cases. Our policy is now defined as
  <span class="math-container">$$\pi(a_t | s_t) = \frac{1}{\sqrt{2\pi \sigma(s_t, \theta)^2}} \exp\left(-\frac{1}{2}\left(\frac{a_t - \mu(s_t, \theta)}{\sigma(s_t, \theta)}\right)^2\right).$$</span></p>
  <p>As you can see, we can easily take the logarithm of this and find the derivative with respect to <span class="math-container">$\theta$</span>, and so nothing changes and the loss you use is the same. You simply evaluate the derivative of the log of your policy with respect to the network parameters, multiply by <span class="math-container">$v_t$</span> and <span class="math-container">$\alpha$</span> and take a gradient step in this direction.</p>
  <p>To implement this (as I'm assuming you don't want to calculate the NN derivatives by hand) then you could do something along the lines of the following in Pytorch.</p>
  <p>First you want to pass your state through your NN to get the mean and standard deviation of the Gaussian distribution. Then you want to simulate <span class="math-container">$z \sim N(0,1)$</span> and calculate <span class="math-container">$a = \mu(s,\theta) + \sigma(s, \theta) \times z$</span> so that <span class="math-container">$a \sim N( \mu(s, \theta), \sigma(s, \theta))$</span> -- this is the reparameterisation trick that makes backpropagation through the network easier as it takes the randomness from a source that doesn't depend on the parameters of the network. <span class="math-container">$a$</span> is your action that you will execute in your environment and use to calculate the gradient by simply writing the code <code>torch.log(normal_pdf(a, \mu(s, \theta), \sigma(s, \theta)).backward()</code> -- here <code>normal_pdf()</code> is any function in Python that calculates the pdf of a normal distribution for a given point and parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-09T22:25:32.070Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118263/discussion-on-answer-by-david-ireland-what-is-the-loss-for-policy-gradients-with).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-31T11:08:20.750Z
Answers.Text: A log-normal distribution is not the same as taking the log of the pdf of the normal distribution, it is taking the log of the random variable.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-31T11:09:10.440Z
Answers.Text: Ah sorry, also just realized that and deleted my comment, but thanks!
Answers.UserDisplayName: Philipp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-01T10:48:38.727Z
Answers.Text: @DavidIreland I modified my neural net, so that the output now has two values (mu and sigma). I saw others implement them in multiple output layer with different activation functions (tanh for mu and softplus for sigma). Is that necessary or can I just have an output layer with two neurons for mu and sigma?
Answers.UserDisplayName: Philipp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-01T10:48:50.557Z
Answers.Text: My main problem however is, that I do not really understand, why you say to just calculate the gradients of the probability density function (pdf; the function for the policy, correct?). Usually I calculate some loss from the output of my neural net and a target output and then call `loss.backward()` and then `optimizer.step()`. How do I include a loss and an optimizer with this probability density function? Could you please explain this? Maybe in an edit to your answer..
Answers.UserDisplayName: Philipp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-01T11:56:23.940Z
Answers.Text: I imagine that when you have seen things like tanh for mu it is for stability. For sigma it is necessary that sigma is non-zero so you will need to force it to be non-zero. I am not an expert numerical stability of neural networks and such but I have had problems in the past, in particular when predicting sigma, is that it is very unstable. What I have seen people do (as usually in ML you end up taking the log of a density) is to directly output the log of the variance, this way you don't need to worry about it being strictly bigger than 0.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-01T12:05:02.317Z
Answers.Text: you can see in the code I attached that I take `backward()` of the log of the pdf. I could similarly have called `loss = torch.log(normal_pdf(a, \mu(s, \theta), \sigma(s, \theta))` and then called `loss.backward()` and this would be using the more traditional variable names. Note that when you do `optimiser.step()` gradient _descent_ is performed so I should technically call `loss = -torch.log(normal_pdf(a, \mu(s, \theta), \sigma(s, \theta))` as then minimising this will be equivalent to maximising the RL objective. Please let me know if this makes sense.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T14:04:07.833Z
Answers.Text: @DavidIreland That makes sense. So we need to take the negative of the log_prob to transform the concave curve (n-shaped) into a convex curve (u-shaped), for which we can find the minimum. This assumes, that the minimum of this curve represents the best mu and the optimal sigma there would be zero, right? Also, why do we need to apply the log to the pdf in ther first place? Wouldn't it be sufficient to calculate the loss as the negative of the pdf itself (given a mu, sigma and action)?
Answers.UserDisplayName: Philipp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T19:55:35.373Z
Answers.Text: @Philipp My advice would be to not think about curve fitting when thinking of RL, it is a very different paradigm to supervised/unsupervised learning. My main advice would be to make sure you go through all the theory (Sutton and Barto is always my go to recommendation) so that you understand why we do what we do. The reason we take the log of the policy is because of how the policy gradient is derived; it is difficult to explain in this little comment section but if you follow the derivation of the policy gradient and how we end up at the REINFORCE update rule it'll make sense.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T20:00:48.927Z
Answers.Text: @Philipp essentially we want to maximise $J(\theta) = v_\pi(s_0)$ where $\theta$ are the policy parameters. We can show that $\nabla_\theta J(\theta) = \mathbb{E}_\mu [ G_t \nabla_\theta \pi_\theta(a|S)]$, where $\mu$ is the state distribution induced by the policy. We can make the expectation also over the action space by writing $\nabla_\theta J(\theta) = \mathbb{E}_{\pi, \mu} [ G_t \frac{\nabla_\theta \pi_\theta(A|S)}{\pi_\theta(A|S)}]$ where the fraction is equal to the $\log$ of the policy, hopefully this clears things up for you :-)
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T06:44:47.673Z
Answers.Text: @DavidIreland If the policy is a Gaussian density function, the gradients have variance in the denominator. How can the algorithm be numerically stable when the learned variance approaches zero?
Answers.UserDisplayName: Cloudy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T09:28:14.077Z
Answers.Text: in practice you would either not learn the variance (because in general learning a variance can be unstable) or you would clip it to make sure it doesn’t ever reach 0.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-01T23:44:31.823Z
Answers.LastActivityDate: 2020-10-02T22:42:06.120Z
Answers.LastEditDate: 2020-10-02T22:42:06.120Z
Answers.UserDisplayName: David
Body: <p>I know with policy gradients used in an environment with a discrete action space are updated with <span class="math-container">$$
  \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}
  $$</span>
  where <span class="math-container">$v_t$</span> could be many things that represent how good the action was. And I know that this can be calculated by performing cross entropy loss with the target being what the network would have outputted if it were completely confident in its action (zeros with the index of the action chosen being one). But I don’t understand how to apply that to policy gradients that output the mean and variance of a Gaussian distribution for a continuous action space. <strong>What is the loss for these types of policy gradients?</strong></p>
  <p>I tried keeping the variance constant and updating the output with mean squared error loss and the target being the action it took. I thought this would end up pushing the mean towards actions with greater total rewards but it got nowhere in OpenAI’s Pendulum environment.</p>
  <p>It would also be very helpful if it was described in a way with a loss function and a target, like how policy gradients with discrete action spaces can be updated with cross entropy loss. That is how I understand it best but it is okay if that is not possible.</p>
  <p><strong>Edit:</strong> for @Philipp. The way I understand it is that the loss function is the same with a continuous action space and the only thing that changes is the distribution that we get the log-probs from. In PyTorch we can use a Normal distribution for continuous action space and Categorical for discrete action space. The answer from David Ireland goes into the math but in PyTorch, that looks like <code>log_prob = distribution.log_prob(action_taken)</code> for any type of distribution. It makes sense that for bad actions we would want to decrease the probability of taking the action. Below is working code for both types of action spaces to compare them. The continuous action space code should be correct but the agent will not learn because it is harder to learn the right actions with a continuous action space and our simple method isn't enough. Look into more advanced methods like <a href="https://spinningup.openai.com/en/latest/algorithms/ppo.html" rel="nofollow noreferrer">PPO</a> and <a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html" rel="nofollow noreferrer">DDPG</a>.</p>
  <pre><code>import torch
  import torch.nn as nn
  import torch.optim as optim
  from torch.distributions.categorical import Categorical #discrete distribution
  import numpy as np
  import gym
  import math
  import matplotlib.pyplot as plt
  
  class Agent(nn.Module):
      def __init__(self,lr):
          super(Agent,self).__init__()
          self.fc1 = nn.Linear(4,64)
          self.fc2 = nn.Linear(64,32)
          self.fc3 = nn.Linear(32,2) #neural network with layers 4,64,32,2
  
          self.optimizer = optim.Adam(self.parameters(),lr=lr)
  
      def forward(self,x):
          x = torch.relu(self.fc1(x)) #relu and tanh for output
          x = torch.relu(self.fc2(x))
          x = torch.sigmoid(self.fc3(x))
          return x
  
  env = gym.make('CartPole-v0')
  agent = Agent(0.001) #hyperparameters
  DISCOUNT = 0.99
  total = []
  
  for e in range(500): 
      log_probs, rewards = [], []
      done = False
      state = env.reset()
      while not done:
          #mu = agent.forward(torch.from_numpy(state).float())
          #distribution = Normal(mu, SIGMA)
          distribution = Categorical(agent.forward(torch.from_numpy(state).float()))
          action = distribution.sample()
          log_probs.append(distribution.log_prob(action))
          state, reward, done, info = env.step(action.item())
          rewards.append(reward)
          
      total.append(sum(rewards))
  
      cumulative = 0
      d_rewards = np.zeros(len(rewards))
      for t in reversed(range(len(rewards))): #get discounted rewards
          cumulative = cumulative * DISCOUNT + rewards[t]
          d_rewards[t] = cumulative
      d_rewards -= np.mean(d_rewards) #normalize
      d_rewards /= np.std(d_rewards)
  
      loss = 0
      for t in range(len(rewards)):
          loss += -log_probs[t] * d_rewards[t] #loss is - log prob * total reward
  
      agent.optimizer.zero_grad()
      loss.backward() #update
      agent.optimizer.step()
  
      if e%10==0:
          print(e,sum(rewards)) 
          plt.plot(total,color='blue') #plot
          plt.pause(0.0001)    
  
  
  def run(i): #to visualize performance
      for _ in range(i):
          done = False
          state = env.reset()
          while not done:
              env.render()
              distribution = Categorical(agent.forward(torch.from_numpy(state).float()))
              action = distribution.sample()
              state,reward,done,info = env.step(action.item())
          env.close()
  </code></pre>
  <p>Above is the discrete action space code for CartPole and below is the continuous action space code for Pendulum. <code>Sigma</code> (variance or standard deviation) is constant here but adding it is easy. Just make the final layer have two neurons and make sure <code>sigma</code> is not negative. Again, the pendulum code won't work because most environments with continuous action spaces are too complicated for such a simple method. Making it work would probably require a lot of testing for hyper parameters.</p>
  <pre><code>import torch
  import torch.nn as nn
  import torch.optim as optim
  from torch.distributions.normal import Normal #continuous distribution
  import numpy as np
  import gym
  import math
  import matplotlib.pyplot as plt
  import keyboard
  
  class Agent(nn.Module):
      def __init__(self,lr):
          super(Agent,self).__init__()
          self.fc1 = nn.Linear(3,64)
          self.fc2 = nn.Linear(64,32)
          self.fc3 = nn.Linear(32,1) #neural network with layers 3,64,32,1
  
          self.optimizer = optim.Adam(self.parameters(),lr=lr)
  
      def forward(self,x):
          x = torch.relu(self.fc1(x)) #relu and tanh for output
          x = torch.relu(self.fc2(x))
          x = torch.tanh(self.fc3(x)) * 2
          return x
  
  env = gym.make('Pendulum-v0')
  agent = Agent(0.01) #hyperparameters
  SIGMA = 0.2
  DISCOUNT = 0.99
  total = []
  
  for e in range(1000): 
      log_probs, rewards = [], []
      done = False
      state = env.reset()
      while not done:
          mu = agent.forward(torch.from_numpy(state).float())
          distribution = Normal(mu, SIGMA)
          action = distribution.sample().clamp(-2.0,2.0)
          log_probs.append(distribution.log_prob(action))
          state, reward, done, info = env.step([action.item()])
          #reward = abs(state[1])
          rewards.append(reward)
          
      total.append(sum(rewards))
  
      cumulative = 0
      d_rewards = np.zeros(len(rewards))
      for t in reversed(range(len(rewards))): #get discounted rewards
          cumulative = cumulative * DISCOUNT + rewards[t]
          d_rewards[t] = cumulative
      d_rewards -= np.mean(d_rewards) #normalize
      d_rewards /= np.std(d_rewards)
  
      loss = 0
      for t in range(len(rewards)):
          loss += -log_probs[t] * d_rewards[t] #loss is - log prob * total reward
  
      agent.optimizer.zero_grad()
      loss.backward() #update
      agent.optimizer.step()
  
      if e%10==0:
          print(e,sum(rewards)) 
          plt.plot(total,color='blue') #plot
          plt.pause(0.0001)
          if keyboard.is_pressed(&quot;space&quot;): #holding space exits training
              raise Exception(&quot;Exited&quot;)
  
  
  def run(i): #to visualize performance
      for _ in range(i):
          done = False
          state = env.reset()
          while not done:
              env.render()
              distribution = Normal(agent.forward(torch.from_numpy(state).float()), SIGMA)
              action = distribution.sample()
              state,reward,done,info = env.step([action.item()])
          env.close()
  </code></pre>
  <p>David Ireland also wrote this on a <a href="https://ai.stackexchange.com/questions/23625/what-exactly-does-the-reinforce-update-equation-mean">different question</a> I had:</p>
  <blockquote>
  <p>The algorithm doesn't change in this situation. Say your NN outputs the mean parameter of the Gaussian, then logπ(at|st) is just the log of the normal density evaluated at the action you took where the mean parameter in the density is the output of your NN. You are then able to backpropagate through this to update the weights of your network.</p>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-30T22:36:13.570Z
Comments.Text: You might want to first express your policy as a density function over the space of possible actions. If you have one of these for each state, and the states are a continuous space as well, you might want to look at Gaussian processes and the likes.
Comments.UserDisplayName: Robby Goetschalckx
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T01:34:56.500Z
Comments.Text: @Robby I don’t understand exactly what you mean. Are you saying I should make the continuous action space discrete? Like if the range of possible actions is -1 to 1 then have '[-1,-0.5,0,0.5,1]' as the only possible actions? I already know how to do that, I am looking for a way to do it with continuous action spaces for problems where that would not work.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T05:49:35.190Z
Comments.Text: No, you don't want it to be discrete. A policy over a continuous space is necessarily a probability density function. With enough training, this should converge to a Dirac delta function centered on the optimal action.
  If you parameterize a density, you can use that in the formula you have, and update the parameters according to the gradient.
Comments.UserDisplayName: Robby Goetschalckx
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T06:49:49.547Z
Comments.Text: @RobbyGoetschalckx I think worth puting in an answer. Probably the most usual way to do this with is have the NN output params of a gaussian - mean and sd (or variance). Showing the gradient terms for mean and sd output when choosing action=x from that distribution should answer OP's question.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T11:39:08.100Z
Comments.Text: @Robby The goal is for it to converge to a Dirac delta function. But what does that update look like with the network outputting the mean and variance of a Gaussian distribution? I am using the Gaussian distribution because it seems like the most common one and after a quick google search I couldn’t find an example of policy gradients with a different distribution function.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T11:39:55.100Z
Comments.Text: @Neil Thanks, that is exactly what I am looking for.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T02:37:14.820Z
Comments.Text: @Philipp I did get it working and I can’t right now but soon I will look at my code to remember how I solved it.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-03T21:52:45.007Z
Comments.Text: @Philipp Sorry it took so long but I finally edited my question to try to help you. I had forgotten a lot of this code. I'm not sure exactly what part you were having trouble with so you can ask more questions if you need. After getting this right, you can code more advanced algorithms like PPO that can actually solve the environments. And I saw the question on your profile, I also thought that I could keep using cross-entropy loss in continuous environments but I don't know if it works because there is not really a clear target.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-04T13:40:38.810Z
Comments.Text: Thank you! This is the most helpful resource on this topic! One question: You calculate a single loss value for every episode based on the cumulative reward at each step and the negative log_prob. Then you back-propagate with this loss value. Could one also use the cumulative reward to find good episodes and then calculate the loss for each step in these good episodes? Then the loss would be just the negative log_prob. I have seen this style with the cross entropy approach, where one would find "elite" episodes and then calculate losses for each step in them. Why didn't you do it like this?
Comments.UserDisplayName: Philipp
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-04T13:52:21.773Z
Comments.Text: @Philipp I didn’t do it like that because I have never heard of it but it sounds interesting. I think it would be a good idea to experiment with. I just learned the method I use and it has worked for me but it sounds like there is no reason your strategy wouldn’t work with a continuous action space if it worked in a discrete action space with cross-entropy. Do you have a link to an example?
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-04T14:01:19.673Z
Comments.Text: @Philipp Maybe we should also talk about this in the [chat](https://chat.stackexchange.com/rooms/122653/loss-for-continous-action-space-comment-discussion) instead. I've never tried this before so it might not work...
Comments.UserDisplayName: S2673
Tags: <neural-networks><reinforcement-learning><policy-gradients><deterministic-policy>
Title: What is the loss for policy gradients with continuous actions?
UserDisplayName: S2673
ThreadText: # What is the loss for policy gradients with continuous actions?
  
  (Asked by: username_0 on 2020-09-30)
  
  I know with policy gradients used in an environment with a discrete action space are updated with $$
  \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}
  $$
  where $v_t$ could be many things that represent how good the action was. And I know that this can be calculated by performing cross entropy loss with the target being what the network would have outputted if it were completely confident in its action (zeros with the index of the action chosen being one). But I don’t understand how to apply that to policy gradients that output the mean and variance of a Gaussian distribution for a continuous action space. What is the loss for these types of policy gradients?
  I tried keeping the variance constant and updating the output with mean squared error loss and the target being the action it took. I thought this would end up pushing the mean towards actions with greater total rewards but it got nowhere in OpenAI’s Pendulum environment.
  It would also be very helpful if it was described in a way with a loss function and a target, like how policy gradients with discrete action spaces can be updated with cross entropy loss. That is how I understand it best but it is okay if that is not possible.
  Edit: for @Philipp. The way I understand it is that the loss function is the same with a continuous action space and the only thing that changes is the distribution that we get the log-probs from. In PyTorch we can use a Normal distribution for continuous action space and Categorical for discrete action space. The answer from David Ireland goes into the math but in PyTorch, that looks like log_prob = distribution.log_prob(action_taken) for any type of distribution. It makes sense that for bad actions we would want to decrease the probability of taking the action. Below is working code for both types of action spaces to compare them. The continuous action space code should be correct but the agent will not learn because it is harder to learn the right actions with a continuous action space and our simple method isn't enough. Look into more advanced methods like PPO and DDPG.
  import torch
  import torch.nn as nn
  import torch.optim as optim
  from torch.distributions.categorical import Categorical #discrete distribution
  import numpy as np
  import gym
  import math
  import matplotlib.pyplot as plt
  
  class Agent(nn.Module):
      def __init__(self,lr):
          super(Agent,self).__init__()
          self.fc1 = nn.Linear(4,64)
          self.fc2 = nn.Linear(64,32)
          self.fc3 = nn.Linear(32,2) #neural network with layers 4,64,32,2
  
          self.optimizer = optim.Adam(self.parameters(),lr=lr)
  
      def forward(self,x):
          x = torch.relu(self.fc1(x)) #relu and tanh for output
          x = torch.relu(self.fc2(x))
          x = torch.sigmoid(self.fc3(x))
          return x
  
  env = gym.make('CartPole-v0')
  agent = Agent(0.001) #hyperparameters
  DISCOUNT = 0.99
  total = []
  
  for e in range(500): 
      log_probs, rewards = [], []
      done = False
      state = env.reset()
      while not done:
          #mu = agent.forward(torch.from_numpy(state).float())
          #distribution = Normal(mu, SIGMA)
          distribution = Categorical(agent.forward(torch.from_numpy(state).float()))
          action = distribution.sample()
          log_probs.append(distribution.log_prob(action))
          state, reward, done, info = env.step(action.item())
          rewards.append(reward)
          
      total.append(sum(rewards))
  
      cumulative = 0
      d_rewards = np.zeros(len(rewards))
      for t in reversed(range(len(rewards))): #get discounted rewards
          cumulative = cumulative * DISCOUNT + rewards[t]
          d_rewards[t] = cumulative
      d_rewards -= np.mean(d_rewards) #normalize
      d_rewards /= np.std(d_rewards)
  
      loss = 0
      for t in range(len(rewards)):
          loss += -log_probs[t] * d_rewards[t] #loss is - log prob * total reward
  
      agent.optimizer.zero_grad()
      loss.backward() #update
      agent.optimizer.step()
  
      if e%10==0:
          print(e,sum(rewards)) 
          plt.plot(total,color='blue') #plot
          plt.pause(0.0001)    
  
  
  def run(i): #to visualize performance
      for _ in range(i):
          done = False
          state = env.reset()
          while not done:
              env.render()
              distribution = Categorical(agent.forward(torch.from_numpy(state).float()))
              action = distribution.sample()
              state,reward,done,info = env.step(action.item())
          env.close()
  
  Above is the discrete action space code for CartPole and below is the continuous action space code for Pendulum. Sigma (variance or standard deviation) is constant here but adding it is easy. Just make the final layer have two neurons and make sure sigma is not negative. Again, the pendulum code won't work because most environments with continuous action spaces are too complicated for such a simple method. Making it work would probably require a lot of testing for hyper parameters.
  import torch
  import torch.nn as nn
  import torch.optim as optim
  from torch.distributions.normal import Normal #continuous distribution
  import numpy as np
  import gym
  import math
  import matplotlib.pyplot as plt
  import keyboard
  
  class Agent(nn.Module):
      def __init__(self,lr):
          super(Agent,self).__init__()
          self.fc1 = nn.Linear(3,64)
          self.fc2 = nn.Linear(64,32)
          self.fc3 = nn.Linear(32,1) #neural network with layers 3,64,32,1
  
          self.optimizer = optim.Adam(self.parameters(),lr=lr)
  
      def forward(self,x):
          x = torch.relu(self.fc1(x)) #relu and tanh for output
          x = torch.relu(self.fc2(x))
          x = torch.tanh(self.fc3(x)) * 2
          return x
  
  env = gym.make('Pendulum-v0')
  agent = Agent(0.01) #hyperparameters
  SIGMA = 0.2
  DISCOUNT = 0.99
  total = []
  
  for e in range(1000): 
      log_probs, rewards = [], []
      done = False
      state = env.reset()
      while not done:
          mu = agent.forward(torch.from_numpy(state).float())
          distribution = Normal(mu, SIGMA)
          action = distribution.sample().clamp(-2.0,2.0)
          log_probs.append(distribution.log_prob(action))
          state, reward, done, info = env.step([action.item()])
          #reward = abs(state[1])
          rewards.append(reward)
          
      total.append(sum(rewards))
  
      cumulative = 0
      d_rewards = np.zeros(len(rewards))
      for t in reversed(range(len(rewards))): #get discounted rewards
          cumulative = cumulative * DISCOUNT + rewards[t]
          d_rewards[t] = cumulative
      d_rewards -= np.mean(d_rewards) #normalize
      d_rewards /= np.std(d_rewards)
  
      loss = 0
      for t in range(len(rewards)):
          loss += -log_probs[t] * d_rewards[t] #loss is - log prob * total reward
  
      agent.optimizer.zero_grad()
      loss.backward() #update
      agent.optimizer.step()
  
      if e%10==0:
          print(e,sum(rewards)) 
          plt.plot(total,color='blue') #plot
          plt.pause(0.0001)
          if keyboard.is_pressed("space"): #holding space exits training
              raise Exception("Exited")
  
  
  def run(i): #to visualize performance
      for _ in range(i):
          done = False
          state = env.reset()
          while not done:
              env.render()
              distribution = Normal(agent.forward(torch.from_numpy(state).float()), SIGMA)
              action = distribution.sample()
              state,reward,done,info = env.step([action.item()])
          env.close()
  
  David Ireland also wrote this on a different question I had:
  
  The algorithm doesn't change in this situation. Say your NN outputs the mean parameter of the Gaussian, then logπ(at|st) is just the log of the normal density evaluated at the action you took where the mean parameter in the density is the output of your NN. You are then able to backpropagate through this to update the weights of your network.
  
  **Tags:** neural-networks, reinforcement-learning, policy-gradients, deterministic-policy
  > **Comments:**
  > **username_1**: You might want to first express your policy as a density function over the space of possible actions. If you have one of these for each state, and the states are a continuous space as well, you might want to look at Gaussian processes and the likes.  
  > **username_0**: @Robby I don’t understand exactly what you mean. Are you saying I should make the continuous action space discrete? Like if the range of possible actions is -1 to 1 then have '[-1,-0.5,0,0.5,1]' as the only possible actions? I already know how to do that, I am looking for a way to do it with continuous action spaces for problems where that would not work.  
  > **username_1**: No, you don't want it to be discrete. A policy over a continuous space is necessarily a probability density function. With enough training, this should converge to a Dirac delta function centered on the optimal action. If you parameterize a density, you can use that in the formula you have, and update the parameters according to the gradient.  
  > **username_2**: @RobbyGoetschalckx I think worth puting in an answer. Probably the most usual way to do this with is have the NN output params of a gaussian - mean and sd (or variance). Showing the gradient terms for mean and sd output when choosing action=x from that distribution should answer OP's question.  
  > **username_0**: @Robby The goal is for it to converge to a Dirac delta function. But what does that update look like with the network outputting the mean and variance of a Gaussian distribution? I am using the Gaussian distribution because it seems like the most common one and after a quick google search I couldn’t find an example of policy gradients with a different distribution function.  
  > **username_0**: @Neil Thanks, that is exactly what I am looking for.  
  > **username_0**: @Philipp I did get it working and I can’t right now but soon I will look at my code to remember how I solved it.  
  > **username_0**: @Philipp Sorry it took so long but I finally edited my question to try to help you. I had forgotten a lot of this code. I'm not sure exactly what part you were having trouble with so you can ask more questions if you need. After getting this right, you can code more advanced algorithms like PPO that can actually solve the environments. And I saw the question on your profile, I also thought that I could keep using cross-entropy loss in continuous environments but I don't know if it works because there is not really a clear target.  
  > **username_3**: Thank you! This is the most helpful resource on this topic! One question: You calculate a single loss value for every episode based on the cumulative reward at each step and the negative log_prob. Then you back-propagate with this loss value. Could one also use the cumulative reward to find good episodes and then calculate the loss for each step in these good episodes? Then the loss would be just the negative log_prob. I have seen this style with the cross entropy approach, where one would find "elite" episodes and then calculate losses for each step in them. Why didn't you do it like this?  
  > **username_0**: @Philipp I didn’t do it like that because I have never heard of it but it sounds interesting. I think it would be a good idea to experiment with. I just learned the method I use and it has worked for me but it sounds like there is no reason your strategy wouldn’t work with a continuous action space if it worked in a discrete action space with cross-entropy. Do you have a link to an example?  
  > **username_0**: @Philipp Maybe we should also talk about this in the [chat](https://chat.stackexchange.com/rooms/122653/loss-for-continous-action-space-comment-discussion) instead. I've never tried this before so it might not work...  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2020-10-01. Score: 3 | Accepted answer
  
  This update rule can still be applied in the continuous domain.
  
  As pointed out in the comments, suppose we are parameterising our policy using a Gaussian distribution, where our neural networks take as input the state we are in and output the parameters of a Gaussian distribution, the mean and the standard deviation which we will denote as $\mu(s, \theta)$ and $\sigma(s, \theta)$ where $s$ shows the dependancy of the state and $\theta$ are the parameters of our network.
  
  I will assume a one-dimensional case for ease of notation but this can be extended to multi-variate cases. Our policy is now defined as
  $$\pi(a\_t | s\_t) = \frac{1}{\sqrt{2\pi \sigma(s\_t, \theta)^2}} \exp\left(-\frac{1}{2}\left(\frac{a\_t - \mu(s\_t, \theta)}{\sigma(s\_t, \theta)}\right)^2\right).$$
  
  As you can see, we can easily take the logarithm of this and find the derivative with respect to $\theta$, and so nothing changes and the loss you use is the same. You simply evaluate the derivative of the log of your policy with respect to the network parameters, multiply by $v\_t$ and $\alpha$ and take a gradient step in this direction.
  
  To implement this (as I'm assuming you don't want to calculate the NN derivatives by hand) then you could do something along the lines of the following in Pytorch.
  
  First you want to pass your state through your NN to get the mean and standard deviation of the Gaussian distribution. Then you want to simulate $z \sim N(0,1)$ and calculate $a = \mu(s,\theta) + \sigma(s, \theta) \times z$ so that $a \sim N( \mu(s, \theta), \sigma(s, \theta))$ -- this is the reparameterisation trick that makes backpropagation through the network easier as it takes the randomness from a source that doesn't depend on the parameters of the network. $a$ is your action that you will execute in your environment and use to calculate the gradient by simply writing the code `torch.log(normal_pdf(a, \mu(s, \theta), \sigma(s, \theta)).backward()` -- here `normal_pdf()` is any function in Python that calculates the pdf of a normal distribution for a given point and parameters.
  
  > **Comments:**
  > **username_5**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118263/discussion-on-answer-by-david-ireland-what-is-the-loss-for-policy-gradients-with).  
  > **username_4**: A log-normal distribution is not the same as taking the log of the pdf of the normal distribution, it is taking the log of the random variable.  
  > **username_3**: Ah sorry, also just realized that and deleted my comment, but thanks!  
  > **username_3**: @DavidIreland I modified my neural net, so that the output now has two values (mu and sigma). I saw others implement them in multiple output layer with different activation functions (tanh for mu and softplus for sigma). Is that necessary or can I just have an output layer with two neurons for mu and sigma?  
  > **username_3**: My main problem however is, that I do not really understand, why you say to just calculate the gradients of the probability density function (pdf; the function for the policy, correct?). Usually I calculate some loss from the output of my neural net and a target output and then call `loss.backward()` and then `optimizer.step()`. How do I include a loss and an optimizer with this probability density function? Could you please explain this? Maybe in an edit to your answer..  
  > **username_4**: I imagine that when you have seen things like tanh for mu it is for stability. For sigma it is necessary that sigma is non-zero so you will need to force it to be non-zero. I am not an expert numerical stability of neural networks and such but I have had problems in the past, in particular when predicting sigma, is that it is very unstable. What I have seen people do (as usually in ML you end up taking the log of a density) is to directly output the log of the variance, this way you don't need to worry about it being strictly bigger than 0.  
  > **username_4**: you can see in the code I attached that I take `backward()` of the log of the pdf. I could similarly have called `loss = torch.log(normal_pdf(a, \mu(s, \theta), \sigma(s, \theta))` and then called `loss.backward()` and this would be using the more traditional variable names. Note that when you do `optimiser.step()` gradient _descent_ is performed so I should technically call `loss = -torch.log(normal_pdf(a, \mu(s, \theta), \sigma(s, \theta))` as then minimising this will be equivalent to maximising the RL objective. Please let me know if this makes sense.  
  > **username_3**: @DavidIreland That makes sense. So we need to take the negative of the log_prob to transform the concave curve (n-shaped) into a convex curve (u-shaped), for which we can find the minimum. This assumes, that the minimum of this curve represents the best mu and the optimal sigma there would be zero, right? Also, why do we need to apply the log to the pdf in ther first place? Wouldn't it be sufficient to calculate the loss as the negative of the pdf itself (given a mu, sigma and action)?  
  > **username_4**: @Philipp My advice would be to not think about curve fitting when thinking of RL, it is a very different paradigm to supervised/unsupervised learning. My main advice would be to make sure you go through all the theory (Sutton and Barto is always my go to recommendation) so that you understand why we do what we do. The reason we take the log of the policy is because of how the policy gradient is derived; it is difficult to explain in this little comment section but if you follow the derivation of the policy gradient and how we end up at the REINFORCE update rule it'll make sense.  
  > **username_4**: @Philipp essentially we want to maximise $J(\theta) = v_\pi(s_0)$ where $\theta$ are the policy parameters. We can show that $\nabla_\theta J(\theta) = \mathbb{E}_\mu [ G_t \nabla_\theta \pi_\theta(a|S)]$, where $\mu$ is the state distribution induced by the policy. We can make the expectation also over the action space by writing $\nabla_\theta J(\theta) = \mathbb{E}_{\pi, \mu} [ G_t \frac{\nabla_\theta \pi_\theta(A|S)}{\pi_\theta(A|S)}]$ where the fraction is equal to the $\log$ of the policy, hopefully this clears things up for you :-)  
  > **username_6**: @DavidIreland If the policy is a Gaussian density function, the gradients have variance in the denominator. How can the algorithm be numerically stable when the learned variance approaches zero?  
  > **username_4**: in practice you would either not learn the variance (because in general learning a variance can be unstable) or you would clip it to make sure it doesn’t ever reach 0.  
  
  ---
  
  