Body: <h3>update</h3>
  <p>After some attempt, I find the learning rate is too small for the version with critic if just adopting the lr(0.0005) of the one without critic. I alter the lr to 0.005 (light red curve) and 0.02(green curve), the return curve goes up but still slow. It is very strange that such big lr is needed! It is way beyond empirical lr range for rl. And why critic need bigger lr than actor?
  <a href="https://i.sstatic.net/30XcW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/30XcW.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/UelF0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UelF0.png" alt="enter image description here" /></a></p>
  <hr />
  <h3>Original question</h3>
  <p>I have implement ppo on my target guard environment. At first I just set critic output to zero and train actor network and get reasonable policy improvement. When I create a critic network and use advantage as the gradient weight, it learns nothing. At first I thought my critic code is messed up but I noticed the explained variance(EV) of my ciritc network is improved, wich means it learns something. Although the EV is not good but at least higher than 0 so it should be better than just gives zeros as the value. And my code is from <a href="https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py" rel="nofollow noreferrer">cleanrl</a>, the critic part is not modified. What might be the reason?</p>
  <h4>Environment description:</h4>
  <p><strong>General</strong>: training a global coordinator to assign 5 defenders to attackers to defend 5 attackers from target area(a circle). one-vs-one interception strategy is predefined.</p>
  <p><strong>Obs</strong>: (5*5, 25) shape, every defender-attaker pair obs consists of [defender_id, attacker_id, aliveness, relative_position, reltive_angle, ...]</p>
  <p><strong>action</strong>: global assignment: [target_id_for_defender_0, target_id_for_defender_1, ...]</p>
  <h4>Critic network architecture:</h4>
  <p>It is verbose to explain it here. Simply put, use self attention to calculate shapley value of every defender-attacker pair and sum up them as the value. More in <a href="https://arxiv.org/abs/2010.13110" rel="nofollow noreferrer">this paper</a></p>
  <p>↓(Blue curve is without critic, red one is with critic.)</p>
  <p><a href="https://i.sstatic.net/19qGq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/19qGq.png" alt="episodic returns" /></a></p>
  <p><a href="https://i.sstatic.net/SzxG6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/SzxG6.png" alt="explained variance" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-14T12:37:17.583Z
Comments.Text: This is likely to be related to the nature of the environment and how you are representing current state. Could you please add some details using [edit]
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-14T12:39:46.047Z
Comments.Text: It may also be related to the critic network architecture, so please share that and the policy network for comparison
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-14T14:50:23.520Z
Comments.Text: @NeilSlater Thanks. I have update my question.
Comments.UserDisplayName: zhixin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-14T16:37:56.777Z
Comments.Text: That critic network seems quite specialised, so it may require someone who knows more about it and your multi-agent use-case, before they can explain your observations about learning rate. Good that you have a work-around and functional Actor-Critic setup now. Although it still seems you get little to no benefit from implementing the critic . . .
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning>
Title: Why my ppo with critic learn nothing while no critic does learns?
UserDisplayName: zhixin
ThreadText: # Why my ppo with critic learn nothing while no critic does learns?
  
  (Asked by: username_0 on 2023-12-14)
  
  ### update
  
  After some attempt, I find the learning rate is too small for the version with critic if just adopting the lr(0.0005) of the one without critic. I alter the lr to 0.005 (light red curve) and 0.02(green curve), the return curve goes up but still slow. It is very strange that such big lr is needed! It is way beyond empirical lr range for rl. And why critic need bigger lr than actor?
  [![enter image description here](https://i.sstatic.net/30XcW.png)](https://i.sstatic.net/30XcW.png)
  [![enter image description here](https://i.sstatic.net/UelF0.png)](https://i.sstatic.net/UelF0.png)
  
  ---
  
  ### Original question
  
  I have implement ppo on my target guard environment. At first I just set critic output to zero and train actor network and get reasonable policy improvement. When I create a critic network and use advantage as the gradient weight, it learns nothing. At first I thought my critic code is messed up but I noticed the explained variance(EV) of my ciritc network is improved, wich means it learns something. Although the EV is not good but at least higher than 0 so it should be better than just gives zeros as the value. And my code is from [cleanrl](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py), the critic part is not modified. What might be the reason?
  
  #### Environment description:
  
  **General**: training a global coordinator to assign 5 defenders to attackers to defend 5 attackers from target area(a circle). one-vs-one interception strategy is predefined.
  
  **Obs**: (5\*5, 25) shape, every defender-attaker pair obs consists of [defender\_id, attacker\_id, aliveness, relative\_position, reltive\_angle, ...]
  
  **action**: global assignment: [target\_id\_for\_defender\_0, target\_id\_for\_defender\_1, ...]
  
  #### Critic network architecture:
  
  It is verbose to explain it here. Simply put, use self attention to calculate shapley value of every defender-attacker pair and sum up them as the value. More in [this paper](https://arxiv.org/abs/2010.13110)
  
  ↓(Blue curve is without critic, red one is with critic.)
  
  [![episodic returns](https://i.sstatic.net/19qGq.png)](https://i.sstatic.net/19qGq.png)
  
  [![explained variance](https://i.sstatic.net/SzxG6.png)](https://i.sstatic.net/SzxG6.png)
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: This is likely to be related to the nature of the environment and how you are representing current state. Could you please add some details using [edit]  
  > **username_1**: It may also be related to the critic network architecture, so please share that and the policy network for comparison  
  > **username_0**: @NeilSlater Thanks. I have update my question.  
  > **username_1**: That critic network seems quite specialised, so it may require someone who knows more about it and your multi-agent use-case, before they can explain your observations about learning rate. Good that you have a work-around and functional Actor-Critic setup now. Although it still seems you get little to no benefit from implementing the critic . . .  
  
  ---
  
  ## 0 answers
  
  