Answers.Body: <p>The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray too far away from the optimal point.</p>
  <p>This means you repeat your step 6. <code>epoch</code> amount of times for the same batch of trajectories. Usually <code>epoch</code> is somewhere between 3 and 30 but it is a hyper-parameter you need to adjust. For the first <em>repeat</em>, the old and the new policy are the same, so their ratio should be 1. After the first update, the new probabilities will change due to the updated policy, whereas you will still need to use the old probabilities giving you a ratio different from 1. The old probabilities will stay the same during these <code>epoch</code> update steps, whereas your new probabilities will keep changing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-27T16:09:39.937Z
Answers.Text: Thank you for your response. When the ratio is 1, how does the new policy change so the ratio isn't one the second time around? Doesn't a ratio of one mean the gradient is zero?
Answers.UserDisplayName: Gideon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T07:56:58.913Z
Answers.Text: @Gideon When the ratio is 1, the update reduces to a simple policy gradient update, for example to one used in the advantage actor-critic. The gradient is not zero, because your parameterized policy $\pi_\theta$ depends on $\theta$ and this gradient will exist. The ratio is just a multiplier telling you how much you can scale the gradient $\partial (\log{\pi_\theta})/\partial \theta$.
Answers.UserDisplayName: Hai Nguyen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T08:02:18.897Z
Answers.Text: If you'd like to "cheat" a bit, you could peek into this repo (https://github.com/seungeunrho/minimalRL/) that contains simple RL implementations in pytorch and compare for example the code for actor-critic and PPO.
Answers.UserDisplayName: Hai Nguyen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-27T07:30:26.420Z
Answers.LastActivityDate: 2020-08-27T07:30:26.420Z
Answers.UserDisplayName: Hai Nguyen
Body: <p>I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target tracking. My question is how the new policy gets updated. My current implementation is as follows</p>
  <ol>
  <li>Make observation (distance from laser to target in pan and tilt)</li>
  <li>Pass observation to actor network which outputs a mean (std for now is fixed)</li>
  <li>I sample from a gaussian with the mean output from step 2</li>
  <li>Apply the command and observe the reward (1/L2 distance to target)</li>
  <li>collect N steps of experience, compute advantage and old log probabilities,</li>
  <li>train actor and critic</li>
  </ol>
  <p>My question is this. I have my old log probabilities (probabilities of the actions taken given the means generated by the actor network), but I dont understand how the new probabilities are generated. At the onset of the very first minibatch my new policy is identical to my old policy as they are the same neural net. Given that in the model.fit function I am passing the same set of observations to generate 'y_pred' values, and I am passing in the actual actions taken as my 'y_true' values, the new policy should generate the exact same log probabilities as my old one. The only (slight) variation that makes the network update is from the entropy bonus, but my score
  np.exp(new_log_probs-old.log_probs)  is nearly identically 1 because the policies are the same.</p>
  <p>Should I be using a pair of networks similar to DDQN so there are some initial differences in the policies between the one used to generate the data and the one used for training?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-26T23:50:57.537Z
Comments.Text: Welcome to SE:AI!  (Thanks for specifying conceptual, which is on-topic. However, if you don't get an answer here, might try CS or Overflow:)
Comments.UserDisplayName: DukeZhou
Tags: <reinforcement-learning><keras><objective-functions><policy-gradients><proximal-policy-optimization>
Title: Generation of 'new log probabilities' in continuous action space PPO
UserDisplayName: Gideon
ThreadText: # Generation of 'new log probabilities' in continuous action space PPO
  
  (Asked by: username_0 on 2020-08-26)
  
  I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target tracking. My question is how the new policy gets updated. My current implementation is as follows
  
  1. Make observation (distance from laser to target in pan and tilt)
  2. Pass observation to actor network which outputs a mean (std for now is fixed)
  3. I sample from a gaussian with the mean output from step 2
  4. Apply the command and observe the reward (1/L2 distance to target)
  5. collect N steps of experience, compute advantage and old log probabilities,
  6. train actor and critic
  
  My question is this. I have my old log probabilities (probabilities of the actions taken given the means generated by the actor network), but I dont understand how the new probabilities are generated. At the onset of the very first minibatch my new policy is identical to my old policy as they are the same neural net. Given that in the model.fit function I am passing the same set of observations to generate 'y\_pred' values, and I am passing in the actual actions taken as my 'y\_true' values, the new policy should generate the exact same log probabilities as my old one. The only (slight) variation that makes the network update is from the entropy bonus, but my score
  np.exp(new\_log\_probs-old.log\_probs) is nearly identically 1 because the policies are the same.
  
  Should I be using a pair of networks similar to DDQN so there are some initial differences in the policies between the one used to generate the data and the one used for training?
  
  **Tags:** reinforcement-learning, keras, objective-functions, policy-gradients, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  (Thanks for specifying conceptual, which is on-topic. However, if you don't get an answer here, might try CS or Overflow:)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-27. Score: 2
  
  The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray too far away from the optimal point.
  
  This means you repeat your step 6. `epoch` amount of times for the same batch of trajectories. Usually `epoch` is somewhere between 3 and 30 but it is a hyper-parameter you need to adjust. For the first *repeat*, the old and the new policy are the same, so their ratio should be 1. After the first update, the new probabilities will change due to the updated policy, whereas you will still need to use the old probabilities giving you a ratio different from 1. The old probabilities will stay the same during these `epoch` update steps, whereas your new probabilities will keep changing.
  
  > **Comments:**
  > **username_0**: Thank you for your response. When the ratio is 1, how does the new policy change so the ratio isn't one the second time around? Doesn't a ratio of one mean the gradient is zero?  
  > **username_2**: @username_0 When the ratio is 1, the update reduces to a simple policy gradient update, for example to one used in the advantage actor-critic. The gradient is not zero, because your parameterized policy $\pi_\theta$ depends on $\theta$ and this gradient will exist. The ratio is just a multiplier telling you how much you can scale the gradient $\partial (\log{\pi_\theta})/\partial \theta$.  
  > **username_2**: If you'd like to "cheat" a bit, you could peek into this repo (https://github.com/seungeunrho/minimalRL/) that contains simple RL implementations in pytorch and compare for example the code for actor-critic and PPO.  
  
  ---
  
  