Answers.Body: <p>A quick recap:</p>
  <ol>
  <li>You train a reinforcement learning agent using PPO</li>
  <li>Your agent learns useful behavior, but this behavior is suboptimal</li>
  <li>You would like your agent to learn a behavior closer to optimal</li>
  </ol>
  <p>Let me suggest four strategies how you can improve you agent's learning. You can
  either use those strategies separately or combined.</p>
  <p>(1) From the evidence you provide (the model does not use the full action parameter range it could use), it seems that the neural network under the hood is not able to leverage the full action range. A useful thing you can do at this stage is to normalize the action space. For instance, assuming you have continuous action environment (<a href="https://github.com/openai/gym/blob/master/gym/wrappers/rescale_action.py" rel="nofollow noreferrer">code source</a>),</p>
  <pre><code>from gym.wrappers import RescaleAction
  
  
  env = RescaleAction(env, min_action=-1, max_action=1)
  </code></pre>
  <p>Now, the neural network will assume that the actions are between -1 and 1, which will likely improve the situation.</p>
  <p>(2) Additionally, to enhance your agent training, you may want to normalize your observations and/or rewards. Here is how you can do it with <a href="https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#stable_baselines3.common.vec_env.VecNormalize" rel="nofollow noreferrer">stable baselines3 vec_env</a>:</p>
  <pre><code>from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize
  
  env = DummyVecEnv([lambda: env])
  
  # Wrap the environment with actions normalization
  env = VecNormalize(env, norm_obs=True, norm_reward=True, clip_obs=10., clip_reward=10., training=True)
  
  model = PPO(MlpPolicy, env, verbose=0)
  </code></pre>
  <p>Note that during evaluation, no need to update the rolling average of
  observations/rewards. This means you will need to set <code>training=False</code> when evaluating agents.</p>
  <p>Try those three normalizations (actions, observations, rewards) separately and then combined and see how does it improve the policy or not.</p>
  <p>(3) If you are designing your own environment, it might be useful to provide the agent more visibility in terms of observation space.
  This is all about feature engineering to provide the agent more information
  about its environment. One example would be including previous observations
  and/or actions into current observations. Note that including too many
  observations can also increase number of training iterations.</p>
  <p>The simplest way to use this approach is by creating a custom <a href="https://www.gymlibrary.dev/api/wrappers/#observationwrapper" rel="nofollow noreferrer">gym wrapper</a>.</p>
  <p>(4) One more thing to do is to find better hyperparameters (e.g. using <a href="https://optuna.org/" rel="nofollow noreferrer">Optuna</a>). Beware, this approach is computationally expensive because for each hyperparameter set it will train a new agent. Given that PPO was created with intention to minimize hyper parameters tuning (I briefly explain the PPO algorithm <a href="https://penkovsky.com/neural-networks/beyond/" rel="nofollow noreferrer">here</a>), hyperparameter search should be used as a last resort only, i.e. after you tried everything else.</p>
  <p>Finally, I have to point out that as soon as neural networks enter the game (as in PPO), there is no guarantee to learn the optimal policy. In case if you solve a simple task, you may want to use simpler algorithms (ones that do not involve neural networks). Here is <a href="http://incompleteideas.net/book/the-book.html" rel="nofollow noreferrer"><em>the</em> book</a> about reinforcement learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T10:36:30.800Z
Answers.LastActivityDate: 2023-08-06T10:44:02.253Z
Answers.LastEditDate: 2023-08-06T10:44:02.253Z
Answers.UserDisplayName: penkovsky
Body: <p>I have a Gym env (env) for which I train with a model using the PPO algorithm with stable-baselines.</p>
  <pre><code>from stable_baselines3 import PPO
  from stable_baselines3.ppo import MlpPolicy
  model = PPO(MlpPolicy, env, verbose=0)
  model.learn(total_timesteps=10_000,progress_bar=True)
  
  obs,info = env.reset()
  action_prediction=[]
  for i in range(5000):
      action, _states = model.predict(obs, deterministic=True)
      action_prediction.append(action)
      obs, rewards, done1, done2, info = env.step(action)
      if done1:
          break
  
  </code></pre>
  <p>If I use the model.predict routine after training, I see that the model successfully masters the task. However, this is not the fastest solution. For the proposed actions, I see for example that the model does not use the full action parameter range it could use. For a test case, where I know the optimal solution, I see that the prediction needs more steps than needed by the optimal solution.</p>
  <p>How can I foster the model during learning to fullfill the task with a minimal number of steps? Is there a better way than going down the limbo of optimizing the reward function to balance mission completion and run time by writing my own Environment?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-01T23:00:50.173Z
Comments.Text: What is the problem that you are trying to solve? Have you considered the possibility that RL might not be the best solution (especially deep RL)? Except for customizing the reward to penalize longer episodes, I see no other way to achieve this when using RL.
Comments.UserDisplayName: pi-tau
Tags: <reinforcement-learning><proximal-policy-optimization><stable-baselines>
Title: Reinforcement learning: Find the fastest solution (minimal number of steps)
UserDisplayName: P. Egli
ThreadText: # Reinforcement learning: Find the fastest solution (minimal number of steps)
  
  (Asked by: username_0 on 2023-08-01)
  
  I have a Gym env (env) for which I train with a model using the PPO algorithm with stable-baselines.
  
  ```
  from stable_baselines3 import PPO
  from stable_baselines3.ppo import MlpPolicy
  model = PPO(MlpPolicy, env, verbose=0)
  model.learn(total_timesteps=10_000,progress_bar=True)
  
  obs,info = env.reset()
  action_prediction=[]
  for i in range(5000):
      action, _states = model.predict(obs, deterministic=True)
      action_prediction.append(action)
      obs, rewards, done1, done2, info = env.step(action)
      if done1:
          break
  
  ```
  
  If I use the model.predict routine after training, I see that the model successfully masters the task. However, this is not the fastest solution. For the proposed actions, I see for example that the model does not use the full action parameter range it could use. For a test case, where I know the optimal solution, I see that the prediction needs more steps than needed by the optimal solution.
  
  How can I foster the model during learning to fullfill the task with a minimal number of steps? Is there a better way than going down the limbo of optimizing the reward function to balance mission completion and run time by writing my own Environment?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, stable-baselines
  
  > **Comments:**
  > **username_1**: What is the problem that you are trying to solve? Have you considered the possibility that RL might not be the best solution (especially deep RL)? Except for customizing the reward to penalize longer episodes, I see no other way to achieve this when using RL.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-06. Score: 0
  
  A quick recap:
  
  1. You train a reinforcement learning agent using PPO
  2. Your agent learns useful behavior, but this behavior is suboptimal
  3. You would like your agent to learn a behavior closer to optimal
  
  Let me suggest four strategies how you can improve you agent's learning. You can
  either use those strategies separately or combined.
  
  (1) From the evidence you provide (the model does not use the full action parameter range it could use), it seems that the neural network under the hood is not able to leverage the full action range. A useful thing you can do at this stage is to normalize the action space. For instance, assuming you have continuous action environment ([code source](https://github.com/openai/gym/blob/master/gym/wrappers/rescale_action.py)),
  
  ```
  from gym.wrappers import RescaleAction
  
  env = RescaleAction(env, min_action=-1, max_action=1)
  
  ```
  
  Now, the neural network will assume that the actions are between -1 and 1, which will likely improve the situation.
  
  (2) Additionally, to enhance your agent training, you may want to normalize your observations and/or rewards. Here is how you can do it with [stable baselines3 vec\_env](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#stable_baselines3.common.vec_env.VecNormalize):
  
  ```
  from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize
  
  env = DummyVecEnv([lambda: env])
  
  # Wrap the environment with actions normalization
  env = VecNormalize(env, norm_obs=True, norm_reward=True, clip_obs=10., clip_reward=10., training=True)
  
  model = PPO(MlpPolicy, env, verbose=0)
  
  ```
  
  Note that during evaluation, no need to update the rolling average of
  observations/rewards. This means you will need to set `training=False` when evaluating agents.
  
  Try those three normalizations (actions, observations, rewards) separately and then combined and see how does it improve the policy or not.
  
  (3) If you are designing your own environment, it might be useful to provide the agent more visibility in terms of observation space.
  This is all about feature engineering to provide the agent more information
  about its environment. One example would be including previous observations
  and/or actions into current observations. Note that including too many
  observations can also increase number of training iterations.
  
  The simplest way to use this approach is by creating a custom [gym wrapper](https://www.gymlibrary.dev/api/wrappers/#observationwrapper).
  
  (4) One more thing to do is to find better hyperparameters (e.g. using [Optuna](https://optuna.org/)). Beware, this approach is computationally expensive because for each hyperparameter set it will train a new agent. Given that PPO was created with intention to minimize hyper parameters tuning (I briefly explain the PPO algorithm [here](https://username_2.com/neural-networks/beyond/)), hyperparameter search should be used as a last resort only, i.e. after you tried everything else.
  
  Finally, I have to point out that as soon as neural networks enter the game (as in PPO), there is no guarantee to learn the optimal policy. In case if you solve a simple task, you may want to use simpler algorithms (ones that do not involve neural networks). Here is [*the* book](http://incompleteideas.net/book/the-book.html) about reinforcement learning.
  
  
  ---
  
  