Answers.Body: <p>The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.</p>
  <p>It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-01T18:51:36.083Z
Answers.Text: I agree. If the algorithm works, then its failure in a different environment is probably because of complexity or rewards.
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T16:15:01.300Z
Answers.Text: Thanks, I tried normalizing my observations between -1 and 1 and this caused a major improvement. I might also try this for the rewards and actions next.
Answers.UserDisplayName: Philipp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-01T17:50:23.423Z
Answers.LastActivityDate: 2021-05-02T12:12:33.993Z
Answers.LastEditDate: 2021-05-02T12:12:33.993Z
Answers.UserDisplayName: Ubikuity
Body: <p>I have built a custom RL environment with <code>gym</code>, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including their state; red, yellow, green). I trained a PPO agent from <code>stable_baselines3</code> without considering setting of hyperparameters and the agent learned to follow the vehicle in front of it without crashing. However it does not learn to stop at red lights after extensive training.</p>
  <p>I tried training it without surrounding vehicles to get more interactions of the RL vehicle with traffic lights and this helped the agent to learn stopping a red light. However when I then continue training of the agent in a new environment with surrounding traffic, the agent again <code>un-learns</code> stopping at red lights.</p>
  <p>I am still a novice with RL and do not understand as to why this happens and what I can do here. Should I set hyperparameters? Or try a different model? Or should I exchange the default policy of the PPO model?</p>
  
Tags: <reinforcement-learning><deep-rl>
Title: PPO agent for vehicle control does not learn to stop at traffic lights
UserDisplayName: Philipp
ThreadText: # PPO agent for vehicle control does not learn to stop at traffic lights
  
  (Asked by: username_0 on 2021-05-01)
  
  I have built a custom RL environment with `gym`, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including their state; red, yellow, green). I trained a PPO agent from `stable_baselines3` without considering setting of hyperparameters and the agent learned to follow the vehicle in front of it without crashing. However it does not learn to stop at red lights after extensive training.
  
  I tried training it without surrounding vehicles to get more interactions of the RL vehicle with traffic lights and this helped the agent to learn stopping a red light. However when I then continue training of the agent in a new environment with surrounding traffic, the agent again `un-learns` stopping at red lights.
  
  I am still a novice with RL and do not understand as to why this happens and what I can do here. Should I set hyperparameters? Or try a different model? Or should I exchange the default policy of the PPO model?
  
  **Tags:** reinforcement-learning, deep-rl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-01. Score: 2 | Accepted answer
  
  The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.
  
  It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.
  
  > **Comments:**
  > **username_2**: I agree. If the algorithm works, then its failure in a different environment is probably because of complexity or rewards.  
  > **username_0**: Thanks, I tried normalizing my observations between -1 and 1 and this caused a major improvement. I might also try this for the rewards and actions next.  
  
  ---
  
  