Body: <p>I built an agent based on PPO with actor-critic structure for a project of path planning, in which the objective of agent is to find a path toward the goal bypass the obstacle in a 16*16 grid map. This is the main information of my model in details:</p>
  <p>Both actor and critic network has the structure of two hidden layers with 128 neuron nodes for each layer and a 40-dim input layer. The 40-dim input is the state observation including two parts of information: the first 32 one-hot generalized elements represent the 2-dim coordinate [x, y], and the last 8 elements represent the obstacle condition of current position (with 1 meaning obstacle and 0 meaning free). the output of actor is an 8-dim vector as the probability of eight actions, whilst that of critic is the state value v(s).<br />
  The main hyperparameters has been set as below:</p>
  <ul>
  <li>Learning rate: actor: 2e-04, cirtic: 4e-04</li>
  <li>Max grad for both networks: 1.5</li>
  <li>Policy clip: epsilon = 0.2</li>
  <li>Discount: gamma = 0.99</li>
  <li>GAE parameter: lambda = 0.95</li>
  <li>Entropy regularization parameter: 0.005</li>
  <li>Batchsize = 128</li>
  </ul>
  <p>The reward function is set like this rules:</p>
  <ul>
  <li>the positive reward (+1.0) is obtained when the manhattan distance from current node to goal is smaller otherwise the negative (-1.0) for larger.</li>
  <li>While the next grid will be the obstacle, the agent will stay with reward 0.0.</li>
  <li>The agent will obtain +5.0 if it leaves from the nodes adjacent the obstacles.</li>
  <li>Reward reshaping: If the agent continuously moves toward the correct direction, there will be an extra adding for rewards.</li>
  </ul>
  <p><a href="https://i.sstatic.net/rjcNn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rjcNn.png" alt="enter image description here" /></a></p>
  <p>Experiment results: In the light of my tracing result for the loss value of both network, I can be quite certain that the model is converging during iteration. However, the convergence does not perform as well as I expected. the MSE loss of critic descent from more than 30.00 at the very beginning but finally stuck around more or less than 10.00 then start to fluctuate. On the other hand, the actor loss is constantly holden around 2.00 and it never be nagative nor has any large changes.</p>
  <p>Moreover, I calculated the average return of the last 50 episodes as the the measurement of convergence. Just as the image illustrating, the return performs fluctuating during the entire training process, meaning it still has no significant improvement after 2,500 training episodes (with 200 maximum step of each epoch). In a quite large probability, the agent still selects the actions to far away the goal or moving to the obstacles.</p>
  <p>Here, I really hope anybody to help me to solve this problem. I will be very grateful for your help and advices. Many thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-20T19:10:19.343Z
Comments.Text: Is the position of the obstacle fixed for this environment, or is it moved in each episode?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-20T19:28:06.737Z
Comments.Text: It may help to watch a learned policy.  There’s a lot of details in the reward structure, plenty of room for misalignment.
Comments.UserDisplayName: foreverska
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-21T12:29:38.643Z
Comments.Text: @Neil Slater
  Sorry for missing for this. Actually, I created several obstacle areas in the map. All of them are static.
Comments.UserDisplayName: speedhawk1
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-21T12:32:50.240Z
Comments.Text: @foreverska
  Thank you for advice. However, could you please explain the 'misalignment' more clearly? Is that meaning the reward function is too complex?
Comments.UserDisplayName: speedhawk1
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-21T16:32:00.907Z
Comments.Text: Misalignment is when AI learns something the designer didn’t intend.  And yes it may be a bit too complex.  Although I’m not well versed in PPO so that’s just a guess.  Like you said, it heads towards obstacles.  This could be because the policy wants to capture that +5 for walking away from it.
Comments.UserDisplayName: foreverska
Tags: <reinforcement-learning><proximal-policy-optimization><path-planning>
Title: PPO agent of path planning does not learn
UserDisplayName: speedhawk1
ThreadText: # PPO agent of path planning does not learn
  
  (Asked by: username_0 on 2024-01-20)
  
  I built an agent based on PPO with actor-critic structure for a project of path planning, in which the objective of agent is to find a path toward the goal bypass the obstacle in a 16\*16 grid map. This is the main information of my model in details:
  
  Both actor and critic network has the structure of two hidden layers with 128 neuron nodes for each layer and a 40-dim input layer. The 40-dim input is the state observation including two parts of information: the first 32 one-hot generalized elements represent the 2-dim coordinate [x, y], and the last 8 elements represent the obstacle condition of current position (with 1 meaning obstacle and 0 meaning free). the output of actor is an 8-dim vector as the probability of eight actions, whilst that of critic is the state value v(s).  
  The main hyperparameters has been set as below:
  
  * Learning rate: actor: 2e-04, cirtic: 4e-04
  * Max grad for both networks: 1.5
  * Policy clip: epsilon = 0.2
  * Discount: gamma = 0.99
  * GAE parameter: lambda = 0.95
  * Entropy regularization parameter: 0.005
  * Batchsize = 128
  
  The reward function is set like this rules:
  
  * the positive reward (+1.0) is obtained when the manhattan distance from current node to goal is smaller otherwise the negative (-1.0) for larger.
  * While the next grid will be the obstacle, the agent will stay with reward 0.0.
  * The agent will obtain +5.0 if it leaves from the nodes adjacent the obstacles.
  * Reward reshaping: If the agent continuously moves toward the correct direction, there will be an extra adding for rewards.
  
  [![enter image description here](https://i.sstatic.net/rjcNn.png)](https://i.sstatic.net/rjcNn.png)
  
  Experiment results: In the light of my tracing result for the loss value of both network, I can be quite certain that the model is converging during iteration. However, the convergence does not perform as well as I expected. the MSE loss of critic descent from more than 30.00 at the very beginning but finally stuck around more or less than 10.00 then start to fluctuate. On the other hand, the actor loss is constantly holden around 2.00 and it never be nagative nor has any large changes.
  
  Moreover, I calculated the average return of the last 50 episodes as the the measurement of convergence. Just as the image illustrating, the return performs fluctuating during the entire training process, meaning it still has no significant improvement after 2,500 training episodes (with 200 maximum step of each epoch). In a quite large probability, the agent still selects the actions to far away the goal or moving to the obstacles.
  
  Here, I really hope anybody to help me to solve this problem. I will be very grateful for your help and advices. Many thanks!
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, path-planning
  
  > **Comments:**
  > **username_1**: Is the position of the obstacle fixed for this environment, or is it moved in each episode?  
  > **username_2**: It may help to watch a learned policy.  There’s a lot of details in the reward structure, plenty of room for misalignment.  
  > **username_0**: @username_1 Sorry for missing for this. Actually, I created several obstacle areas in the map. All of them are static.  
  > **username_0**: @username_2 Thank you for advice. However, could you please explain the 'misalignment' more clearly? Is that meaning the reward function is too complex?  
  > **username_2**: Misalignment is when AI learns something the designer didn’t intend.  And yes it may be a bit too complex.  Although I’m not well versed in PPO so that’s just a guess.  Like you said, it heads towards obstacles.  This could be because the policy wants to capture that +5 for walking away from it.  
  
  ---
  
  ## 0 answers
  
  