Answers.Body: <p>Human performance on Breakout is ~30, if you refer to the <a href="https://arxiv.org/pdf/1312.5602.pdf" rel="nofollow noreferrer">original DQN paper</a> (Table 1). In the <a href="https://arxiv.org/pdf/1602.01783.pdf" rel="nofollow noreferrer">original A3C paper</a>, it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3).</p>
  <p>Is the total loss going down in your original experiment? If you set the entropy coefficient to zero, then the contribution of the entropy loss to the overall loss is zero'ed out. That's why it's actually expected that the entropy loss won't decrease.</p>
  <p>You're taking the gradients of
  <span class="math-container">$$L_{\text{total}}(\theta_{\text{policy}}, \theta_{\text{value}}, \theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}}) - 0 * L_{\text{entropy}}(\theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}})$$</span> where <span class="math-container">$\theta_{\text{policy}}$</span> are the parameters of the policy network, etc.</p>
  <p>Therefore, <span class="math-container">$L_{\text{entropy}}(\theta_{\text{entropy}})$</span> can be anything, it's unconstrained by the optimization of <span class="math-container">$L_{\text{total}}$</span></p>
  <p>To test that you don't have a bug, you should in fact increase the entropy coefficient and check that the entropy loss is decreasing.</p>
  <p>As a reminder, the entropy loss is actually here to maximize the entropy of the policy so that it doesn't collapse to a deterministic one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T20:01:33.993Z
Answers.LastActivityDate: 2022-06-15T20:01:33.993Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m steps the model has not made any significant progress. Im using baseline hyperparameters which can be found here <a href="https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py" rel="nofollow noreferrer">https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py</a>, except my buffersize have been from 512 to 4096.  Ive noticed that entropy decreases extremely slowly</p>
  <p><a href="https://i.sstatic.net/ahGMN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ahGMN.png" alt="enter image description here" /></a></p>
  <p>given the buffersize from the interval which i just gave. So my questions are how to make entropy decrease and how to increase rewards per buffer?  Ive tried to decrease the entropy coefficient to almost zero, but still it acts very weirdly.</p>
  <p>Update: Even when i set the entropy coef to zero entropy wont decrease, i guess i might have a bug?</p>
  <p><a href="https://i.sstatic.net/p7D2h.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/p7D2h.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/Frt4L.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Frt4L.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><deep-rl><advantage-actor-critic>
Title: How to tune hypeparametes in A2C-ppo?
UserDisplayName: voroshilov
ThreadText: # How to tune hypeparametes in A2C-ppo?
  
  (Asked by: username_0 on 2022-06-14)
  
  Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m steps the model has not made any significant progress. Im using baseline hyperparameters which can be found here <https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py>, except my buffersize have been from 512 to 4096. Ive noticed that entropy decreases extremely slowly
  
  [![enter image description here](https://i.sstatic.net/ahGMN.png)](https://i.sstatic.net/ahGMN.png)
  
  given the buffersize from the interval which i just gave. So my questions are how to make entropy decrease and how to increase rewards per buffer? Ive tried to decrease the entropy coefficient to almost zero, but still it acts very weirdly.
  
  Update: Even when i set the entropy coef to zero entropy wont decrease, i guess i might have a bug?
  
  [![enter image description here](https://i.sstatic.net/p7D2h.png)](https://i.sstatic.net/p7D2h.png)
  
  [![enter image description here](https://i.sstatic.net/Frt4L.png)](https://i.sstatic.net/Frt4L.png)
  
  **Tags:** reinforcement-learning, deep-rl, advantage-actor-critic
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-15. Score: 0 | Accepted answer
  
  Human performance on Breakout is ~30, if you refer to the [original DQN paper](https://arxiv.org/pdf/1312.5602.pdf) (Table 1). In the [original A3C paper](https://arxiv.org/pdf/1602.01783.pdf), it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3).
  
  Is the total loss going down in your original experiment? If you set the entropy coefficient to zero, then the contribution of the entropy loss to the overall loss is zero'ed out. That's why it's actually expected that the entropy loss won't decrease.
  
  You're taking the gradients of
  $$L\_{\text{total}}(\theta\_{\text{policy}}, \theta\_{\text{value}}, \theta\_{\text{entropy}}) = L\_{\text{policy gradient}}(\theta\_{\text{policy}}) + c \* L\_{\text{value}}(\theta\_{\text{value}}) - 0 \* L\_{\text{entropy}}(\theta\_{\text{entropy}}) = L\_{\text{policy gradient}}(\theta\_{\text{policy}}) + c \* L\_{\text{value}}(\theta\_{\text{value}})$$ where $\theta\_{\text{policy}}$ are the parameters of the policy network, etc.
  
  Therefore, $L\_{\text{entropy}}(\theta\_{\text{entropy}})$ can be anything, it's unconstrained by the optimization of $L\_{\text{total}}$
  
  To test that you don't have a bug, you should in fact increase the entropy coefficient and check that the entropy loss is decreasing.
  
  As a reminder, the entropy loss is actually here to maximize the entropy of the policy so that it doesn't collapse to a deterministic one.
  
  
  ---
  
  