Answers.Body: <p>I know little to nothing about your problem, but from your code it seems that you use NN to predict the parameters of a Dirichlet distribution... however, i'm not really sure about which entropy you are considering... is it the entropy with respect the sampled discrete distribution from the Dirichlet, or the entropy of the Dirichlet itself</p>
  <p>The problem is that it's very important that initially the agent uses a uniform distribution (discrete one), so it would be better to have a &quot;biased&quot; Dirichlet, so that uniform distribution are more likely than skewed ones... however, this implies that the entropy of the Dirichlet is very low, and might happens that with time it increases...</p>
  <p>Now given this, if you instead start with a uniform Dirichlet, at the beginning of the training you might sample &quot;non-sense&quot; skewed distribution, just because your agent is not calibrated (so the high prob of an action is not due to learning, but to ill-calibration), so entropy is low (for a wrong reason) and then the agent might learn that he prefers more uniform distribution, increasing the entropy</p>
  <p>So my takeaway would be to bias the initialization of the NN so that outputs <span class="math-container">$\alpha$</span> that prefers uniform distributions than skewed ones</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T08:57:23.333Z
Answers.Text: Thanks for your reply.
  I think I implemented what you proposed, without solving the problem. I initialized the actor network weights in the following way:  
  
  def init_weights(self):
          for m in self.modules():
              if isinstance(m, nn.Linear):
                  torch.nn.init.normal_(m.weight, mean=0, std=0.01)
                  m.bias.data.fill_(0.01)
  
  Was that what you meant?
Answers.UserDisplayName: GPra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T09:05:23.790Z
Answers.Text: Here the results of the training. the critic's loss remains a little lower but the problem is not solved.
  <https://imgur.com/wWdmBMU>
Answers.UserDisplayName: GPra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T09:59:40.703Z
Answers.Text: @GPra I don't really have much experience with dirichlet distributions, but it seems that if you want to bias your network to prefer uniform distributions, you have to use high $\alpha$ https://en.wikipedia.org/wiki/File:Dirichlet.pdf
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T10:53:27.860Z
Answers.Text: okay, I think I understand better now, basically I have to initialize the network with a bias value and the higher the bias value, the higher the alpha value and therefore the more uniform the dirichlet will be. 
  Because of the exp activation α=exp(bias)? Is that correct?
Answers.UserDisplayName: GPra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T11:15:58.727Z
Answers.Text: @GPra "_more uniform the dirichlet will be_" not really, if you have high $\alpha$, the dirichlet will be like a "gaussian centered in the middle of the simplex", so the samples will be more uniform distributions... question, why are you using dirichlet instead of a categorical distribution?
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T11:37:42.853Z
Answers.Text: I am working in a continuous action space, whose sum of the 3 actions must be 1. I chose Dirichlet because it seemed to me to be the distribution that best fit to represent my problem and that also gave the best results in the initial experiments, with a version of the environment that was more simplified than the current one and that worked perfectly
Answers.UserDisplayName: GPra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T11:50:44.970Z
Answers.Text: _continuous action space, whose sum of the 3 actions must be 1_ isn't this the definition of a categorical distribution? I just saying because it's much simpler to calibrate, and since Dir dist are used as categorical prior, I thought you were in the setting of discrete actions
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T10:10:00.810Z
Answers.Text: https://www.sciencedirect.com/science/article/pii/S0951832022001831
Answers.UserDisplayName: GPra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T11:26:39.540Z
Answers.Text: @GPra personally I find that they are cherry picking the examples... indeed, if you use a softmax activation over samples from Gaussian, you can have the variance they are saying... but the point is that you can just sample from the softmax, not having such problem (as done in the rest of the literature - that single paper)...
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T13:27:23.293Z
Answers.Text: Perhaps I have not explained myself or I am missing some fundamental point. In my problem, at each step, the agent receives a variable number of requests it has to decide how many to process locally, how many to forward and how many to reject, and not whether it should process them locally, forward or reject. For example, an action could be [0.7, 0.2, 0.1], which means 70 locally, 20 forward and 10 reject. it seems to me a continuous space, not discrete. Excuse me if I am not clear or if I don't understand something you are telling me.
Answers.UserDisplayName: GPra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T15:52:36.383Z
Answers.Text: @GPra when we say continuous/discrete, we refer to the support, not the values... every common distribution is continuous in their values, but some have a discrete support (for example in your case, you have a discrete number of channels)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T01:03:35.387Z
Answers.LastActivityDate: 2023-09-05T01:03:35.387Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>At a glance my guess is that what you are observing should be nothing to worry about.</p>
  <p>For the critic loss what you have to consider is that as the size of the values grows i.e. the agent gets better the scale of errors will grow too because there are simply more ways to be wrong about predicting a value between 0 and 1000 than between 0 and 10. as an example of what you should be looking at as a performance metric is something like symmetric percentage error or explained variance like in the stable baselines implementation.</p>
  <p>For the entropy loss something similar is happening. As the probability distribution becomes more concentrated entropy loss will <strong>increase</strong>. This is because the entropy of a 1-hot distribution is 0 and as it tends towards a uniform distribution that value increases however in standard PPO you subtract the entropy from the loss to incentivise higher entropy. You may have noticed that that would imply growing negative values but you are using differential entropy because of the dirichlet distribution instead of discrete entropy for a categorical which means that the entropy values you can get can be negative. In essence the higher the entropy loss the more deterministic your policy not less.</p>
  <p>In short you've done a great job and have nothing to worry about.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-06T19:57:36.103Z
Answers.LastActivityDate: 2024-11-06T19:57:36.103Z
Answers.UserDisplayName: Leonhard Piff
Body: <p>I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here <a href="https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src" rel="nofollow noreferrer">https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src</a>.</p>
  <p>My doubts are mainly due to the convergence of entropy and critic network loss.
  Over a long training (1M steps), the loss of the critical network grows over time, parallel to the loss of entropy, that it grows too. The growth in entropy is indicative of the agent taking increasingly random actions.</p>
  <p>Despite this, the reward definitely improves as episodes pass.
  It seems that as the critic's loss and entropy increase, the reward improves accordingly. It is as if as the randomness of the actions taken increases, the reward the agent gets also improves.</p>
  <p>This obviously does not make much sense and I would like to understand the reasons for this situation.</p>
  <p><a href="https://i.sstatic.net/9IyPD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9IyPD.png" alt="enter image description here" /></a></p>
  <p>The thing that worries me is that it all seems strongly correlated:</p>
  <ul>
  <li>entropy grows, critic's loss grows, reward grows</li>
  <li>entropy decreases, critic's loss decreases, reward decreases</li>
  </ul>
  <p>My fear is that if entropy and loss converge, reward would collapse, at least that is the feeling I get from seeing the training trend.
  mi sembra che non stia capendo che sta ottendno una buona policy
  it seems to me that he is not understanding that he is getting a good policy.</p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><proximal-policy-optimization>
Title: Is my PPO agent learning? or is it just exploring?
UserDisplayName: GPra
ThreadText: # Is my PPO agent learning? or is it just exploring?
  
  (Asked by: username_0 on 2023-09-04)
  
  I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here <https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src>.
  
  My doubts are mainly due to the convergence of entropy and critic network loss.
  Over a long training (1M steps), the loss of the critical network grows over time, parallel to the loss of entropy, that it grows too. The growth in entropy is indicative of the agent taking increasingly random actions.
  
  Despite this, the reward definitely improves as episodes pass.
  It seems that as the critic's loss and entropy increase, the reward improves accordingly. It is as if as the randomness of the actions taken increases, the reward the agent gets also improves.
  
  This obviously does not make much sense and I would like to understand the reasons for this situation.
  
  [![enter image description here](https://i.sstatic.net/9IyPD.png)](https://i.sstatic.net/9IyPD.png)
  
  The thing that worries me is that it all seems strongly correlated:
  
  * entropy grows, critic's loss grows, reward grows
  * entropy decreases, critic's loss decreases, reward decreases
  
  My fear is that if entropy and loss converge, reward would collapse, at least that is the feeling I get from seeing the training trend.
  mi sembra che non stia capendo che sta ottendno una buona policy
  it seems to me that he is not understanding that he is getting a good policy.
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, proximal-policy-optimization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-09-05. Score: 1
  
  I know little to nothing about your problem, but from your code it seems that you use NN to predict the parameters of a Dirichlet distribution... however, i'm not really sure about which entropy you are considering... is it the entropy with respect the sampled discrete distribution from the Dirichlet, or the entropy of the Dirichlet itself
  
  The problem is that it's very important that initially the agent uses a uniform distribution (discrete one), so it would be better to have a "biased" Dirichlet, so that uniform distribution are more likely than skewed ones... however, this implies that the entropy of the Dirichlet is very low, and might happens that with time it increases...
  
  Now given this, if you instead start with a uniform Dirichlet, at the beginning of the training you might sample "non-sense" skewed distribution, just because your agent is not calibrated (so the high prob of an action is not due to learning, but to ill-calibration), so entropy is low (for a wrong reason) and then the agent might learn that he prefers more uniform distribution, increasing the entropy
  
  So my takeaway would be to bias the initialization of the NN so that outputs $\alpha$ that prefers uniform distributions than skewed ones
  
  > **Comments:**
  > **username_0**: Thanks for your reply. I think I implemented what you proposed, without solving the problem. I initialized the actor network weights in the following way:    def init_weights(self):         for m in self.modules():             if isinstance(m, nn.Linear):                 torch.nn.init.normal_(m.weight, mean=0, std=0.01)                 m.bias.data.fill_(0.01)  Was that what you meant?  
  > **username_0**: Here the results of the training. the critic's loss remains a little lower but the problem is not solved. <https://imgur.com/wWdmBMU>  
  > **username_1**: @username_0 I don't really have much experience with dirichlet distributions, but it seems that if you want to bias your network to prefer uniform distributions, you have to use high $\alpha$ https://en.wikipedia.org/wiki/File:Dirichlet.pdf  
  > **username_0**: okay, I think I understand better now, basically I have to initialize the network with a bias value and the higher the bias value, the higher the alpha value and therefore the more uniform the dirichlet will be.  Because of the exp activation α=exp(bias)? Is that correct?  
  > **username_1**: @username_0 "_more uniform the dirichlet will be_" not really, if you have high $\alpha$, the dirichlet will be like a "gaussian centered in the middle of the simplex", so the samples will be more uniform distributions... question, why are you using dirichlet instead of a categorical distribution?  
  > **username_0**: I am working in a continuous action space, whose sum of the 3 actions must be 1. I chose Dirichlet because it seemed to me to be the distribution that best fit to represent my problem and that also gave the best results in the initial experiments, with a version of the environment that was more simplified than the current one and that worked perfectly  
  > **username_1**: _continuous action space, whose sum of the 3 actions must be 1_ isn't this the definition of a categorical distribution? I just saying because it's much simpler to calibrate, and since Dir dist are used as categorical prior, I thought you were in the setting of discrete actions  
  > **username_0**: https://www.sciencedirect.com/science/article/pii/S0951832022001831  
  > **username_1**: @username_0 personally I find that they are cherry picking the examples... indeed, if you use a softmax activation over samples from Gaussian, you can have the variance they are saying... but the point is that you can just sample from the softmax, not having such problem (as done in the rest of the literature - that single paper)...  
  > **username_0**: Perhaps I have not explained myself or I am missing some fundamental point. In my problem, at each step, the agent receives a variable number of requests it has to decide how many to process locally, how many to forward and how many to reject, and not whether it should process them locally, forward or reject. For example, an action could be [0.7, 0.2, 0.1], which means 70 locally, 20 forward and 10 reject. it seems to me a continuous space, not discrete. Excuse me if I am not clear or if I don't understand something you are telling me.  
  > **username_1**: @username_0 when we say continuous/discrete, we refer to the support, not the values... every common distribution is continuous in their values, but some have a discrete support (for example in your case, you have a discrete number of channels)  
  
  ---
  
  ### Answer by username_2 on 2024-11-06. Score: 0
  
  At a glance my guess is that what you are observing should be nothing to worry about.
  
  For the critic loss what you have to consider is that as the size of the values grows i.e. the agent gets better the scale of errors will grow too because there are simply more ways to be wrong about predicting a value between 0 and 1000 than between 0 and 10. as an example of what you should be looking at as a performance metric is something like symmetric percentage error or explained variance like in the stable baselines implementation.
  
  For the entropy loss something similar is happening. As the probability distribution becomes more concentrated entropy loss will **increase**. This is because the entropy of a 1-hot distribution is 0 and as it tends towards a uniform distribution that value increases however in standard PPO you subtract the entropy from the loss to incentivise higher entropy. You may have noticed that that would imply growing negative values but you are using differential entropy because of the dirichlet distribution instead of discrete entropy for a categorical which means that the entropy values you can get can be negative. In essence the higher the entropy loss the more deterministic your policy not less.
  
  In short you've done a great job and have nothing to worry about.
  
  
  ---
  
  