Answers.Body: <p>Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (and substantially faster training) due to the extra baseline and also the bootstrapping which is done in the generalized advantage estimation (GAE). I don't see how the episode length being short is relevant.</p>
  <p>However, if you don't want to use an advantage, you don't have to. You can simply replace <span class="math-container">$A_t$</span> with the (monte carlo sampled) cumulative reward <span class="math-container">$R_t$</span>. The problem is that the clipping objective only makes sense if you use an advantage. It won't make sense to do that clipping if you are using <span class="math-container">$R_t$</span> instead of <span class="math-container">$A_t$</span>.</p>
  <p>It seems dubious to me to call such an algorithm &quot;PPO&quot; as really it would just be REINFORCE. The only possible difference between what you describe and reinforce, is that PPO should generate a dataset of transitions and do multiple mini-batch updates per dataset. If you do decide you want to go that route, you should note that you still need the importance sampling term (the unfortunately named <span class="math-container">$r_t(\theta)$</span>) for the update to be unbiased.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T15:36:56.203Z
Answers.Text: The real reason why I don't want to use a critic is because I am using a specific NN architecture called Neural Logic Machine and it might be difficult to predict $V(s)$. As you mention, the reason why I want to use PPO instead of REINFORCE is to be able to increase the sample-efficiency of REINFORCE, by performing multiple gradient ascent iterations over the same data. However, if by using $R_t$ instead of the advantage then the clipping objective does not make sense, does this mean I could change the policy probabilities too much by performing multiple training iterations over the same data?
Answers.UserDisplayName: Aeryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T16:12:17.880Z
Answers.Text: Yes it's possible that the training will be a bit more unstable without the clipping. But my biggest concern is that without the bootstrapping (which is included in gae) the algorithm may learn quite slowly
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T16:43:16.813Z
Answers.Text: So, if I have understood you correctly, you think that even if the trajectories are short I should try to use actor-critic instead of vanilla REINFORCE and that will make a bigger impact in the sample-efficiency of my algorithm than using PPO (instead of REINFORCE) and performing multiple gradient ascents over the same data.
Answers.UserDisplayName: Aeryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T17:20:24.123Z
Answers.Text: I am saying that the generalized advantage estimator (GAE) seems to be really good and I would highly recommend using it for the advantage calculation; yes this would require having a critic in addition to the policy. The question of sample efficiency is separate and it's hard to make any quantifiable claim there. But yes I believe it will definitely help.
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T01:55:10.677Z
Answers.LastActivityDate: 2022-06-12T01:55:10.677Z
Answers.UserDisplayName: Taw
Body: <p>I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my solution. From the original paper, it seems the clipped loss PPO optimizes includes an Advantage <span class="math-container">$A_t$</span> term:</p>
  <p><span class="math-container">$$
  L^{CLIP} (\theta) = \mathbb{E}_t \Big[min(r_t(\theta)A_t, 
  \: clip(r_t(\theta), 1 - \epsilon, 1 + \epsilon)A_t \Big]
  $$</span></p>
  <p>Generally, this advantage <span class="math-container">$A_t$</span> is calculated using an <em>actor-critic</em> schema, where we train a network (critic) to predict the value <span class="math-container">$V(s)$</span> of a given state, which is then used to calculate <span class="math-container">$A_t$</span>. However, my RL task is episodic and the trajectories end (i.e., arrive at a terminal state) in just a few actions, so I would rather not use a critic network. Thus, my question is the following: <strong>How can I implement PPO without a critic network, i.e., without a network that predicts <span class="math-container">$V(s)$</span>?</strong></p>
  <p>To achieve this, I thought about simply substituting the advantage term <span class="math-container">$A_t$</span> used in <span class="math-container">$L^{CLIP}$</span> for the discounted sum of rewards <span class="math-container">$R_t$</span> used in the REINFORCE loss. <strong>Should this work fine or is there a better alternative?</strong></p>
  
Tags: <deep-rl><actor-critic-methods><proximal-policy-optimization><reinforce>
Title: How to implement PPO without using a Critic
UserDisplayName: Aeryan
ThreadText: # How to implement PPO without using a Critic
  
  (Asked by: username_0 on 2022-06-11)
  
  I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my solution. From the original paper, it seems the clipped loss PPO optimizes includes an Advantage $A\_t$ term:
  
  $$
  L^{CLIP} (\theta) = \mathbb{E}\_t \Big[min(r\_t(\theta)A\_t,
  \: clip(r\_t(\theta), 1 - \epsilon, 1 + \epsilon)A\_t \Big]
  $$
  
  Generally, this advantage $A\_t$ is calculated using an *actor-critic* schema, where we train a network (critic) to predict the value $V(s)$ of a given state, which is then used to calculate $A\_t$. However, my RL task is episodic and the trajectories end (i.e., arrive at a terminal state) in just a few actions, so I would rather not use a critic network. Thus, my question is the following: **How can I implement PPO without a critic network, i.e., without a network that predicts $V(s)$?**
  
  To achieve this, I thought about simply substituting the advantage term $A\_t$ used in $L^{CLIP}$ for the discounted sum of rewards $R\_t$ used in the REINFORCE loss. **Should this work fine or is there a better alternative?**
  
  **Tags:** deep-rl, actor-critic-methods, proximal-policy-optimization, reinforce
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-12. Score: 1 | Accepted answer
  
  Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (and substantially faster training) due to the extra baseline and also the bootstrapping which is done in the generalized advantage estimation (GAE). I don't see how the episode length being short is relevant.
  
  However, if you don't want to use an advantage, you don't have to. You can simply replace $A\_t$ with the (monte carlo sampled) cumulative reward $R\_t$. The problem is that the clipping objective only makes sense if you use an advantage. It won't make sense to do that clipping if you are using $R\_t$ instead of $A\_t$.
  
  It seems dubious to me to call such an algorithm "PPO" as really it would just be REINFORCE. The only possible difference between what you describe and reinforce, is that PPO should generate a dataset of transitions and do multiple mini-batch updates per dataset. If you do decide you want to go that route, you should note that you still need the importance sampling term (the unfortunately named $r\_t(\theta)$) for the update to be unbiased.
  
  > **Comments:**
  > **username_0**: The real reason why I don't want to use a critic is because I am using a specific NN architecture called Neural Logic Machine and it might be difficult to predict $V(s)$. As you mention, the reason why I want to use PPO instead of REINFORCE is to be able to increase the sample-efficiency of REINFORCE, by performing multiple gradient ascent iterations over the same data. However, if by using $R_t$ instead of the advantage then the clipping objective does not make sense, does this mean I could change the policy probabilities too much by performing multiple training iterations over the same data?  
  > **username_1**: Yes it's possible that the training will be a bit more unstable without the clipping. But my biggest concern is that without the bootstrapping (which is included in gae) the algorithm may learn quite slowly  
  > **username_0**: So, if I have understood you correctly, you think that even if the trajectories are short I should try to use actor-critic instead of vanilla REINFORCE and that will make a bigger impact in the sample-efficiency of my algorithm than using PPO (instead of REINFORCE) and performing multiple gradient ascents over the same data.  
  > **username_1**: I am saying that the generalized advantage estimator (GAE) seems to be really good and I would highly recommend using it for the advantage calculation; yes this would require having a critic in addition to the policy. The question of sample efficiency is separate and it's hard to make any quantifiable claim there. But yes I believe it will definitely help.  
  
  ---
  
  