Answers.Body: <blockquote>
    <p>Do neural networks compute the probability distribution for policy gradient methods. </p>
  </blockquote>
  
  <p>In short, yes. It does not have to be neural networks, any trainable parametric function approximator based on gradients will do. Neural networks are a common choice, as are linear function approximators using selected basis functions.</p>
  
  <blockquote>
    <p>If so, how do they compute an infinite probability distribution? </p>
  </blockquote>
  
  <p>For background, this is an issue for generating stochastic policies in continuous action spaces only. In discrete action spaces, it is usually possible to compute an arbitrary probability density function for the whole action space, and sample from it to model the policy. It is also possible to compute a deterministic policy simply enough in continuous spaces - the input is the current state and the output is the action to take. The issue then is that this does not allow an agent to learn through exploration of the environment. To do that requires a stochastic policy.</p>
  
  <p>If you want to generate a stochastic policy in continuous action spaces, you could discretise the space and sample from that using e.g. softmax to generate the action probabilities. Or you could have the approximation function do something more indirect: Output the parameters of a probability distribution that can be sampled from.</p>
  
  <blockquote>
    <p>How do you represent a continuous action policy with a neural network?</p>
  </blockquote>
  
  <p>Typically by having state features as input and the parameters of a PDF that can be sampled as the output. For instance, the network could output mean <span class="math-container">$\mu$</span> and standard deviation <span class="math-container">$\sigma$</span> of a normal distribution for an action value, and the policy is given by <span class="math-container">$\pi(a|s) = a \sim \mathbb{N}(\mu, \sigma)$</span>. </p>
  
  <p>This distribution can be sampled (there are simple methods to generate a sample from a normal distribution), and returns from following this policy used as feedback to the neural network using the policy gradient theorem. Assuming that there is an optimal deterministic policy to be found, the neural network can learn over time to home in on a specific mean with low standard deviation.</p>
  
  <p>In some cases, the standard deviation can be treated as a hyper-parameter, similar to <span class="math-container">$\epsilon$</span> in <span class="math-container">$\epsilon$</span>-greedy action selection, and might be decayed over time. In that case, a neural network can just output the mean action.</p>
  
  <p>It is also possible to learn a deterministic policy through off-policy learning, adding a noise function to support exploration. This is what Deep Deterministic Policy Gradient does.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-16T06:58:07.077Z
Answers.LastActivityDate: 2019-09-16T08:26:34.763Z
Answers.LastEditDate: 2019-09-16T08:26:34.763Z
Answers.UserDisplayName: Neil Slater
Body: <p>Do neural networks compute the probability distribution for policy gradient methods. If so, how do they compute an infinite probability distribution? How do you represent a continuous action policy with a neural network?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-15T21:50:31.680Z
Comments.Text: Hi and welcome to this community! Can you be more specific: which probability distribution are you referring to?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-15T21:53:42.737Z
Comments.Text: I'm not an expert and I don't know much about it, but for a continuous action space, don't stochastic policy gradients compute a probability for performing each action, so if there are infinite actions, then the model will have to compute an infinite amount of probabilities
Comments.UserDisplayName: Josh Goldman
Tags: <neural-networks><reinforcement-learning><policy-gradients>
Title: How do policy gradients compute an infinite probability distribution from a neural network
UserDisplayName: Josh Goldman
ThreadText: # How do policy gradients compute an infinite probability distribution from a neural network
  
  (Asked by: username_0 on 2019-09-15)
  
  Do neural networks compute the probability distribution for policy gradient methods. If so, how do they compute an infinite probability distribution? How do you represent a continuous action policy with a neural network?
  
  **Tags:** neural-networks, reinforcement-learning, policy-gradients
  
  > **Comments:**
  > **username_1**: Hi and welcome to this community! Can you be more specific: which probability distribution are you referring to?  
  > **username_0**: I'm not an expert and I don't know much about it, but for a continuous action space, don't stochastic policy gradients compute a probability for performing each action, so if there are infinite actions, then the model will have to compute an infinite amount of probabilities  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-16. Score: 1
  
  > Do neural networks compute the probability distribution for policy gradient methods.
  
  In short, yes. It does not have to be neural networks, any trainable parametric function approximator based on gradients will do. Neural networks are a common choice, as are linear function approximators using selected basis functions.
  
  > If so, how do they compute an infinite probability distribution?
  
  For background, this is an issue for generating stochastic policies in continuous action spaces only. In discrete action spaces, it is usually possible to compute an arbitrary probability density function for the whole action space, and sample from it to model the policy. It is also possible to compute a deterministic policy simply enough in continuous spaces - the input is the current state and the output is the action to take. The issue then is that this does not allow an agent to learn through exploration of the environment. To do that requires a stochastic policy.
  
  If you want to generate a stochastic policy in continuous action spaces, you could discretise the space and sample from that using e.g. softmax to generate the action probabilities. Or you could have the approximation function do something more indirect: Output the parameters of a probability distribution that can be sampled from.
  
  > How do you represent a continuous action policy with a neural network?
  
  Typically by having state features as input and the parameters of a PDF that can be sampled as the output. For instance, the network could output mean $\mu$ and standard deviation $\sigma$ of a normal distribution for an action value, and the policy is given by $\pi(a|s) = a \sim \mathbb{N}(\mu, \sigma)$.
  
  This distribution can be sampled (there are simple methods to generate a sample from a normal distribution), and returns from following this policy used as feedback to the neural network using the policy gradient theorem. Assuming that there is an optimal deterministic policy to be found, the neural network can learn over time to home in on a specific mean with low standard deviation.
  
  In some cases, the standard deviation can be treated as a hyper-parameter, similar to $\epsilon$ in $\epsilon$-greedy action selection, and might be decayed over time. In that case, a neural network can just output the mean action.
  
  It is also possible to learn a deterministic policy through off-policy learning, adding a noise function to support exploration. This is what Deep Deterministic Policy Gradient does.
  
  
  ---
  
  