Answers.Body: <p>You can check the Open AI <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#id14" rel="nofollow noreferrer">Introduction to RL series</a>, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE algorithm, we are talking about the one described in <a href="https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf" rel="nofollow noreferrer">Sutton's book on Reinforcement learning</a>. It is described as the policy optimization algorithm maximizing the Value Function <span class="math-container">$v_{\pi(\theta)}(s) = E[G_t|S_t = s]$</span> of initial state of the agent. Here <span class="math-container">$G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$</span> is the <span class="math-container">$\gamma$</span> discounted return from given state, time  <span class="math-container">$s, t$</span>. Or, shortly put.
  <span class="math-container">$$
   J(\theta) = v_{\pi(\theta)}(s_0) = E[G_t|S_t = s_0]\\
  \nabla J(\theta) = E_\pi\left[G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}\right]
  $$</span>
  But in the RL series of Open AI, the algorithm that is described as Vanilla policy gradient (If it is the one you are talking about) is optimizing finite-horizon undiscounted return <span class="math-container">$E_{\tau \sim \pi} [R(\tau)] $</span>, where <span class="math-container">$\tau$</span> are possible trajectories. e.g.
  <span class="math-container">$$
   J(\theta) =  E_{\tau \sim \pi} [R(\tau)] \\
  \nabla J(\theta) = E_{\tau \sim\pi}\left[\sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}\right]
  $$</span></p>
  <p><strong>They do look very similar in objective functions, but they are different.</strong> The way the gradient ascent is performed differs strongly since in REINFORCE method the gradient ascent is performed once for each action taken for each episode and the direction of ascent is taken as</p>
  <p><span class="math-container">$$
  G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
  $$</span>
  so the update becomes
  <span class="math-container">$$
  \theta_{t+1} = \theta_{t} + \alpha G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
  $$</span>
  but in VPG algorithm the gradient ascents performed once over multiple episodes and direction of ascent taken as average</p>
  <p><span class="math-container">$$
  \frac{1}{|\mathcal{T}|}\sum_{\tau\in\mathcal{T}} \sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
  $$</span></p>
  <p>and gradient ascent step is</p>
  <p><span class="math-container">$$
  \theta_{t+1} = \theta_{t} + \alpha \frac{1}{|\mathcal{T}|}\sum_{\tau\in\mathcal{T}} \sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
  $$</span>
  which looks a lot like what you have stated as REINFORCE algorithm.</p>
  <p>I admit, that some form of mathematical equivalence can be derived between them, since expectation over policy and expectation over the trajectory sampled from the policy looks practically the same. But approaches differ at least in the way, the ascent is computed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-13T03:42:48.870Z
Answers.Text: the answer is great.
Answers.UserDisplayName: DehengYe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-30T13:44:13.317Z
Answers.LastActivityDate: 2022-09-09T12:52:47.360Z
Answers.LastEditDate: 2022-09-09T12:52:47.360Z
Answers.UserDisplayName: vl_knd
Answers.Body: <p>OpenAI has a series of Spinning Up pages on their website to educate people about AI. One of those defines Vanilla Policy Gradiant.</p>
  
  <p><a href="https://spinningup.openai.com/en/latest/algorithms/vpg.html" rel="nofollow noreferrer">Vanilla Policy Gradiant via OpenAI</a></p>
  
  <p>At the bottom of the page are reference papers that further discuss gradiants. </p>
  
  <p>Whether this is definitive for Vanilla Policy Gradients or not I do not know, but if many others refer to OpenAI for learning this subject their definition will spread.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T21:22:45.747Z
Answers.LastActivityDate: 2019-05-31T21:22:45.747Z
Answers.UserDisplayName: jeffry7
Answers.Body: <p>By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization (or whatever is trying to be communicated).</p>
  <p>For example, let <span class="math-container">$J(\theta)$</span> be any policy objective function. Then our policy gradient would be <span class="math-container">$\begin{equation} \nabla_\theta J(\theta)  \end{equation}$</span> where the change in our policy is <span class="math-container">$\Delta \theta = \alpha \nabla_\theta J(\theta)  $</span> with <span class="math-container">$\alpha$</span> being our step size.</p>
  <p>This will, of course, vary as notational convention and term definition can change between practitioners.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T09:48:05.583Z
Answers.LastActivityDate: 2022-01-12T21:09:58.670Z
Answers.LastEditDate: 2022-01-12T21:09:58.670Z
Answers.UserDisplayName: hisairnessag3
Body: <p>I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?</p>
  <p>By REINFORCE I mean this surrogate objective</p>
  <p><span class="math-container">$$ \frac{1}{m} \sum_i \sum_t log(\pi(a_t|s_t)) R_i,$$</span>
  where I index over the <span class="math-container">$m$</span> episodes and <span class="math-container">$t$</span> over time steps, and <span class="math-container">$R_i$</span> is the total reward of the episode. It's also common to replace <span class="math-container">$R_i$</span> with something else, like a baselined version <span class="math-container">$R_i - b$</span> or use the future return, potentially also with a baseline <span class="math-container">$G_{it} - b$</span>.</p>
  <p>However, I think even with these modifications to the multiplicative term, people would still call this 'vanilla policy gradient'. Is that correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-27T13:43:51.147Z
Comments.Text: Use of the term may vary, depending on context, and what "extra" feature is being discussed. Could you give a reference to where someone uses "vanilla policy gradient"? With nothing else to go on, I think your understanding seems reasonable.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-27T14:53:24.510Z
Comments.Text: I'm thinking of TRPO and PPO right now, where the 'vanilla' part often implies that it's not sample efficient because it only performs one optimization step with the data. I can't point to other specific papers right now, but I think I've seen the term 'vanilla policy gradient' in other contexts too. Are there any other aspects of REINFORCE besides the single-optimization-update that makes it 'vanilla'?
Comments.UserDisplayName: yewang
Tags: <reinforcement-learning><comparison><terminology><policy-gradients><reinforce>
Title: Is REINFORCE the same as 'vanilla policy gradient'?
UserDisplayName: yewang
ThreadText: # Is REINFORCE the same as 'vanilla policy gradient'?
  
  (Asked by: username_0 on 2019-03-27)
  
  I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?
  
  By REINFORCE I mean this surrogate objective
  
  $$ \frac{1}{m} \sum\_i \sum\_t log(\pi(a\_t|s\_t)) R\_i,$$
  where I index over the $m$ episodes and $t$ over time steps, and $R\_i$ is the total reward of the episode. It's also common to replace $R\_i$ with something else, like a baselined version $R\_i - b$ or use the future return, potentially also with a baseline $G\_{it} - b$.
  
  However, I think even with these modifications to the multiplicative term, people would still call this 'vanilla policy gradient'. Is that correct?
  
  **Tags:** reinforcement-learning, comparison, terminology, policy-gradients, reinforce
  > **Comments:**
  > **username_1**: Use of the term may vary, depending on context, and what "extra" feature is being discussed. Could you give a reference to where someone uses "vanilla policy gradient"? With nothing else to go on, I think your understanding seems reasonable.  
  > **username_0**: I'm thinking of TRPO and PPO right now, where the 'vanilla' part often implies that it's not sample efficient because it only performs one optimization step with the data. I can't point to other specific papers right now, but I think I've seen the term 'vanilla policy gradient' in other contexts too. Are there any other aspects of REINFORCE besides the single-optimization-update that makes it 'vanilla'?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2022-01-30. Score: 6
  
  You can check the Open AI [Introduction to RL series](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#id14), they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE algorithm, we are talking about the one described in [Sutton's book on Reinforcement learning](https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf). It is described as the policy optimization algorithm maximizing the Value Function $v\_{\pi(\theta)}(s) = E[G\_t|S\_t = s]$ of initial state of the agent. Here $G\_t = \sum\_{k=0}^\infty \gamma^k R\_{t+k+1}$ is the $\gamma$ discounted return from given state, time $s, t$. Or, shortly put.
  $$
  J(\theta) = v\_{\pi(\theta)}(s\_0) = E[G\_t|S\_t = s\_0]\\
  \nabla J(\theta) = E\_\pi\left[G\_t\frac{\nabla \pi (A\_t |S\_t, \theta)}{\pi (A\_t |S\_t, \theta)}\right]
  $$
  But in the RL series of Open AI, the algorithm that is described as Vanilla policy gradient (If it is the one you are talking about) is optimizing finite-horizon undiscounted return $E\_{\tau \sim \pi} [R(\tau)] $, where $\tau$ are possible trajectories. e.g.
  $$
  J(\theta) = E\_{\tau \sim \pi} [R(\tau)] \\
  \nabla J(\theta) = E\_{\tau \sim\pi}\left[\sum\_{t=0}^T R(\tau) \frac{\nabla \pi (A\_t |S\_t, \theta)}{\pi (A\_t |S\_t, \theta)}\right]
  $$
  
  **They do look very similar in objective functions, but they are different.** The way the gradient ascent is performed differs strongly since in REINFORCE method the gradient ascent is performed once for each action taken for each episode and the direction of ascent is taken as
  
  $$
  G\_t\frac{\nabla \pi (A\_t |S\_t, \theta)}{\pi (A\_t |S\_t, \theta)}
  $$
  so the update becomes
  $$
  \theta\_{t+1} = \theta\_{t} + \alpha G\_t\frac{\nabla \pi (A\_t |S\_t, \theta)}{\pi (A\_t |S\_t, \theta)}
  $$
  but in VPG algorithm the gradient ascents performed once over multiple episodes and direction of ascent taken as average
  
  $$
  \frac{1}{|\mathcal{T}|}\sum\_{\tau\in\mathcal{T}} \sum\_{t=0}^T R(\tau) \frac{\nabla \pi (A\_t |S\_t, \theta)}{\pi (A\_t |S\_t, \theta)}
  $$
  
  and gradient ascent step is
  
  $$
  \theta\_{t+1} = \theta\_{t} + \alpha \frac{1}{|\mathcal{T}|}\sum\_{\tau\in\mathcal{T}} \sum\_{t=0}^T R(\tau) \frac{\nabla \pi (A\_t |S\_t, \theta)}{\pi (A\_t |S\_t, \theta)}
  $$
  which looks a lot like what you have stated as REINFORCE algorithm.
  
  I admit, that some form of mathematical equivalence can be derived between them, since expectation over policy and expectation over the trajectory sampled from the policy looks practically the same. But approaches differ at least in the way, the ascent is computed.
  
  > **Comments:**
  > **username_3**: the answer is great.  
  
  ---
  
  ### Answer by username_4 on 2019-05-31. Score: 1
  
  OpenAI has a series of Spinning Up pages on their website to educate people about AI. One of those defines Vanilla Policy Gradiant.
  
  [Vanilla Policy Gradiant via OpenAI](https://spinningup.openai.com/en/latest/algorithms/vpg.html)
  
  At the bottom of the page are reference papers that further discuss gradiants.
  
  Whether this is definitive for Vanilla Policy Gradients or not I do not know, but if many others refer to OpenAI for learning this subject their definition will spread.
  
  
  ---
  
  ### Answer by username_5 on 2019-03-28. Score: 1
  
  By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization (or whatever is trying to be communicated).
  
  For example, let $J(\theta)$ be any policy objective function. Then our policy gradient would be $\begin{equation} \nabla\_\theta J(\theta) \end{equation}$ where the change in our policy is $\Delta \theta = \alpha \nabla\_\theta J(\theta) $ with $\alpha$ being our step size.
  
  This will, of course, vary as notational convention and term definition can change between practitioners.
  
  
  ---
  
  