Answers.Body: <p>Policy gradient methods directly learn parameters of a policy function, which is a mapping from states to actions. For example, <span class="math-container">$p(s, a)$</span> can denote a function which takes a state <span class="math-container">$s$</span> and an action <span class="math-container">$a$</span> as input, and returns a probability of taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> as output (equivalently it could just take <span class="math-container">$s$</span> as input, and output a vector or a distribution of probabilities for all actions). Such a function will often have some parameters (for example, a vector of weights if it's a linear function of the state, or matrices of weights in the case of a neural network). Policy gradient methods directly modify the function by modifying (learning) the parameters of that function.</p>
  
  <p>The Bellman equation is typically used to learn a function that computes different kinds of values, such as <span class="math-container">$Q(s, a)$</span> values, which can intuitively be understood to denote all the future rewards that we expect to get if we take action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> (and follow a certain policy, typically "the optimal policy" afterwards until the end of time). Such a <span class="math-container">$Q(s, a)$</span> function that we learn is not a directly learned policy, because it is not a direct mapping from current states to actions, it doesn't directly tell us which action to take.</p>
  
  <p>It can still be used indirectly to create a policy. For example, once such a <span class="math-container">$Q(s, a)$</span> function is learned, the greedy policy can be easily implemented by looping over all possible actions, and taking the action that gives the greatest <span class="math-container">$Q(s, a)$</span> output. This is indirect because the learned function does not directly give you the actions, but only provides information which you can convert into actions with some extra processing.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-19T00:30:26.780Z
Answers.Text: This makes sense. So, in theory: Q-learning is more general? I've heard that policy gradient methods are sample efficent. But Q-Learning methods are more expressive. Would you agree?
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-19T08:38:26.433Z
Answers.Text: I suppose you could say Q-learning is more general in the sense that it's off-policy yes, which means that you can follow a certain policy (e.g. epsilon-greedy), and learn the Q-values for a different policy (e.g. the greedy policy). This is not possible with policy gradient methods as far as I'm aware, they're always on-policy, meaning they optimize their actions for their current policy. This typically results in slightly ''safer'' behaviour if you're using an epsilon-greedy policy for example, because it takes into account that it will sometimes select completely random actions
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-19T08:39:58.363Z
Answers.Text: Q-learning is indeed typically less sample efficient, because it cannot learn from the n-step return after a sequence of n steps if anywhere within that sequence an action was taken that would not have been taken by the policy you're learning values for. That's in the typical case though, there are ways to improve this (e.g. Retrace(lambda))
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-12-18T14:20:04.910Z
Answers.LastActivityDate: 2020-04-03T19:50:19.863Z
Answers.LastEditDate: 2020-04-03T19:50:19.863Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I was watching a lecture on policy gradients and Bellman equations. And they say that a Bellman equation indirectly creates a policy, while the policy gradient directly learns a policy. Why is this?</p>
  
Tags: <reinforcement-learning><comparison><policy-gradients><bellman-equations>
Title: Why do Bellman equations indirectly create a policy?
UserDisplayName: echo
ThreadText: # Why do Bellman equations indirectly create a policy?
  
  (Asked by: username_0 on 2017-12-18)
  
  I was watching a lecture on policy gradients and Bellman equations. And they say that a Bellman equation indirectly creates a policy, while the policy gradient directly learns a policy. Why is this?
  
  **Tags:** reinforcement-learning, comparison, policy-gradients, bellman-equations
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-12-18. Score: 2 | Accepted answer
  
  Policy gradient methods directly learn parameters of a policy function, which is a mapping from states to actions. For example, $p(s, a)$ can denote a function which takes a state $s$ and an action $a$ as input, and returns a probability of taking action $a$ in state $s$ as output (equivalently it could just take $s$ as input, and output a vector or a distribution of probabilities for all actions). Such a function will often have some parameters (for example, a vector of weights if it's a linear function of the state, or matrices of weights in the case of a neural network). Policy gradient methods directly modify the function by modifying (learning) the parameters of that function.
  
  The Bellman equation is typically used to learn a function that computes different kinds of values, such as $Q(s, a)$ values, which can intuitively be understood to denote all the future rewards that we expect to get if we take action $a$ in state $s$ (and follow a certain policy, typically "the optimal policy" afterwards until the end of time). Such a $Q(s, a)$ function that we learn is not a directly learned policy, because it is not a direct mapping from current states to actions, it doesn't directly tell us which action to take.
  
  It can still be used indirectly to create a policy. For example, once such a $Q(s, a)$ function is learned, the greedy policy can be easily implemented by looping over all possible actions, and taking the action that gives the greatest $Q(s, a)$ output. This is indirect because the learned function does not directly give you the actions, but only provides information which you can convert into actions with some extra processing.
  
  > **Comments:**
  > **username_0**: This makes sense. So, in theory: Q-learning is more general? I've heard that policy gradient methods are sample efficent. But Q-Learning methods are more expressive. Would you agree?  
  > **username_1**: I suppose you could say Q-learning is more general in the sense that it's off-policy yes, which means that you can follow a certain policy (e.g. epsilon-greedy), and learn the Q-values for a different policy (e.g. the greedy policy). This is not possible with policy gradient methods as far as I'm aware, they're always on-policy, meaning they optimize their actions for their current policy. This typically results in slightly ''safer'' behaviour if you're using an epsilon-greedy policy for example, because it takes into account that it will sometimes select completely random actions  
  > **username_1**: Q-learning is indeed typically less sample efficient, because it cannot learn from the n-step return after a sequence of n steps if anywhere within that sequence an action was taken that would not have been taken by the policy you're learning values for. That's in the typical case though, there are ways to improve this (e.g. Retrace(lambda))  
  
  ---
  
  