Answers.Body: <p>In the context of RL, for a policy to be parameterised it typically means we explicitly model the policy and is common in policy gradient methods. </p>
  
  <p>Consider value based methods such as Q-learning where our policy is usually something like <span class="math-container">$\epsilon$</span>-greedy where we choose our action using the following policy</p>
  
  <p><span class="math-container">\begin{align}
  \pi(a|s) = \left\{ \begin{array}{ll}
  \arg \max_a Q(s,a) &amp; \text{with probability } 1-\epsilon\;; \\ \text{random action} &amp; \text{with probability } \epsilon\;.
  \end{array}\right.
  \end{align}</span>
  Here we have parameterised the policy with <span class="math-container">$\epsilon$</span> but the learning is done by learning the Q-functions. When we parameterise a policy we will explicitly model <span class="math-container">$\pi$</span> by the following: 
  <span class="math-container">$$\pi(s|a,\boldsymbol{\theta}) = \mathbb{P}(A_t = a | S_t=s, \boldsymbol{\theta}_t = \boldsymbol{\theta})\;.$$</span>
  Learning is now done by learning the parameter <span class="math-container">$\boldsymbol{\theta}$</span> that <em>maximise</em> some performance measure <span class="math-container">$J(\boldsymbol{\theta})$</span> by doing approximate gradient ascent updates of the form
  <span class="math-container">$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t + \alpha \hat{\Delta J(\boldsymbol{\theta}_t)}. $$</span></p>
  
  <p>Note that, as per the Sutton and Barto textbook, <span class="math-container">$\hat{\Delta J(\boldsymbol{\theta}_t)}$</span> is a noisy, stochastic estimate of <span class="math-container">$\Delta J(\boldsymbol{\theta}_t)$</span> where the former approximates the latter in expectation. </p>
  
  <p>The policy can be parameterised in any way as long as it is differentiable with respect to the parameters. Commonly in Deep RL the policy is parameterised as a neural network so <span class="math-container">$\boldsymbol{\theta}$</span> would be the weights of the network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T14:19:02.627Z
Answers.LastActivityDate: 2020-05-14T14:19:02.627Z
Answers.UserDisplayName: David
Body: <p>Can you explain policy gradient methods and what it means for the policy to be parameterised? I am reading Sutton and Barto book on reinforcement learning and didn't understand well what it is, can you give some examples?</p>
  
Tags: <reinforcement-learning><policy-gradients>
Title: What does it mean to parameterise a policy in policy gradient methods?
UserDisplayName: dato nefaridze
ThreadText: # What does it mean to parameterise a policy in policy gradient methods?
  
  (Asked by: username_0 on 2020-05-06)
  
  Can you explain policy gradient methods and what it means for the policy to be parameterised? I am reading Sutton and Barto book on reinforcement learning and didn't understand well what it is, can you give some examples?
  
  **Tags:** reinforcement-learning, policy-gradients
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-14. Score: 2
  
  In the context of RL, for a policy to be parameterised it typically means we explicitly model the policy and is common in policy gradient methods.
  
  Consider value based methods such as Q-learning where our policy is usually something like $\epsilon$-greedy where we choose our action using the following policy
  
  \begin{align}
  \pi(a|s) = \left\{ \begin{array}{ll}
  \arg \max\_a Q(s,a) & \text{with probability } 1-\epsilon\;; \\ \text{random action} & \text{with probability } \epsilon\;.
  \end{array}\right.
  \end{align}
  Here we have parameterised the policy with $\epsilon$ but the learning is done by learning the Q-functions. When we parameterise a policy we will explicitly model $\pi$ by the following:
  $$\pi(s|a,\boldsymbol{\theta}) = \mathbb{P}(A\_t = a | S\_t=s, \boldsymbol{\theta}\_t = \boldsymbol{\theta})\;.$$
  Learning is now done by learning the parameter $\boldsymbol{\theta}$ that *maximise* some performance measure $J(\boldsymbol{\theta})$ by doing approximate gradient ascent updates of the form
  $$\boldsymbol{\theta}\_{t+1} = \boldsymbol{\theta}\_t + \alpha \hat{\Delta J(\boldsymbol{\theta}\_t)}. $$
  
  Note that, as per the Sutton and Barto textbook, $\hat{\Delta J(\boldsymbol{\theta}\_t)}$ is a noisy, stochastic estimate of $\Delta J(\boldsymbol{\theta}\_t)$ where the former approximates the latter in expectation.
  
  The policy can be parameterised in any way as long as it is differentiable with respect to the parameters. Commonly in Deep RL the policy is parameterised as a neural network so $\boldsymbol{\theta}$ would be the weights of the network.
  
  
  ---
  
  