Answers.Body: <p>You can use <a href="https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html" rel="nofollow noreferrer">Maskable PPO</a>, which is PPO but with support for invalid actions. In your case, you need to implement the method <code>action_masks()</code> by yourself, which returns <code>True</code> or <code>False</code> depending whether the action is invalid.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-02T14:45:17.613Z
Answers.LastActivityDate: 2023-03-02T14:45:17.613Z
Answers.UserDisplayName: Minh-Long Luu
Body: <p>I'm training a FPS agent using StableBaselines 3's PPO algorithm.</p>
  <p>To aid learning, I would like to train the agent using just a basic set of actions (e.g Turn left, turn right, shoot). After the agent has mastered that scenario, I'd want to expand the action space with the ability to move forward.</p>
  <p>Additional actions would also be included later on, such as moving left, and right.</p>
  <p>PPO has a check for matching action spaces, so just providing a new environment to the model doesn't work.</p>
  <p>My final question is:
  How can I expand the action space so that the PPO model doesn't reject the new agent?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-02T21:34:53.327Z
Comments.Text: Hi @Ilija Vuk and welcome to AI Stack Exchange! This post seems to contain multiple good questions, especially the 3 question at the end of the post. Typically, this stack exchange website aims to have a single question per post. If possible, please edit this post to a single question, and possibly ask other questions in a separate post. This will help the post conform to this site's guidelines and will probably help you receive an answer faster. Thank you for posting, and we are looking forward to more of your questions on this site!
Comments.UserDisplayName: DeepQZero
Tags: <proximal-policy-optimization><stable-baselines>
Title: How to expand an agent's action space with more actions?
UserDisplayName: Ilija Vuk
ThreadText: # How to expand an agent's action space with more actions?
  
  (Asked by: username_0 on 2023-03-02)
  
  I'm training a FPS agent using StableBaselines 3's PPO algorithm.
  
  To aid learning, I would like to train the agent using just a basic set of actions (e.g Turn left, turn right, shoot). After the agent has mastered that scenario, I'd want to expand the action space with the ability to move forward.
  
  Additional actions would also be included later on, such as moving left, and right.
  
  PPO has a check for matching action spaces, so just providing a new environment to the model doesn't work.
  
  My final question is:
  How can I expand the action space so that the PPO model doesn't reject the new agent?
  
  **Tags:** proximal-policy-optimization, stable-baselines
  
  > **Comments:**
  > **username_1**: Hi @username_0 and welcome to AI Stack Exchange! This post seems to contain multiple good questions, especially the 3 question at the end of the post. Typically, this stack exchange website aims to have a single question per post. If possible, please edit this post to a single question, and possibly ask other questions in a separate post. This will help the post conform to this site's guidelines and will probably help you receive an answer faster. Thank you for posting, and we are looking forward to more of your questions on this site!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-03-02. Score: 0 | Accepted answer
  
  You can use [Maskable PPO](https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html), which is PPO but with support for invalid actions. In your case, you need to implement the method `action_masks()` by yourself, which returns `True` or `False` depending whether the action is invalid.
  
  
  ---
  
  