Body: <p>I am applying PPO to a custom environment and it struggles when the optimal action seems to be at the boundary of the action space. I replicated it in the following simple environment. There are two state variables: time and a variable x, and at each step the reward is -(x+a)^2 + 1, where a is a parameter. At each step the agent should choose x for the next period. x must be in a certain interval. Totally run 10 periods.</p>
  <p>The theoretical optimal policy should be just to choose -a at each step. But what I have found is that when the optimal solution is close to the boundary, for example when period reward is -(x+0.9)^2 + 1 and the constraint for x is [-1,1], PPO at best can give me a policy that picks -1 at each period. PPO can solve the problem though, when the solution is &quot;interior&quot; enough (e.g., with reward -x^2 + 1), and it converges to a very deterministic policy.</p>
  <p>I looked into what is going on during the training, and it seems that when solution is near the boundary, the policy never goes down to a very deterministic one (std of the Gaussian policy remains at a relatively high level, ~0.3-0.4), and hence it just cannot settle down. In fact, mean_reward does not fully converge (as it should have, in such a simple case).</p>
  <p>I can kind of see why this happens, but I do not fully understand what feature of PPO lead to this, and how to fix it. I am also surprised that the optimal solution is not that near the boundary yet my PPO agent cannot find it. Is this something that can be fixed via hyperparameter tuning (if so, how?), or is there any other algorithm better for this kind of tasks (I wanted to stick to PPO because it works for the more complicated task that I am actually trying to solve)?</p>
  <p>I am using stable-baseline 3. Here is a minimum script that can re-produce the problem</p>
  <pre><code>import numpy as np
  import gymnasium as gym
  from gymnasium import spaces
  from stable_baselines3 import PPO
  from stable_baselines3.common.evaluation import evaluate_policy
  from stable_baselines3.common.callbacks import EvalCallback
  from stable_baselines3.common.utils import set_random_seed
  from stable_baselines3.common.env_util import make_vec_env
  import os
  import torch
  
  # Setting up the environment
  home_dir = os.path.expanduser(&quot;~&quot;)
  dir = home_dir+''
  GLOBAL_SEED = 0
  set_random_seed(GLOBAL_SEED) # This is to ensure reproducibility
  #%%
  class SimpleOptEnv(gym.Env):
      def __init__(self, max_steps=10):
          super(SimpleOptEnv, self).__init__()
          self.action_space = spaces.Box(low=-1, high=1, shape=(1,), dtype=np.float32)
          self.observation_space = spaces.Box(low=-1, high=1, shape=(2,), dtype=np.float32)
          self.max_steps = max_steps
          self.current_step = 0
          self.state = None
  
      def reset(self, seed=GLOBAL_SEED, options=None):
          super().reset(seed=seed)
          self.current_step = 0
          self.state = -0.9
          return np.array([self.state, self.current_step / self.max_steps], dtype=np.float32), {}
  
      def step(self, action):
          self.current_step += 1
          reward = -1 * (self.state+0.9)**2 + 1
          self.state = np.clip(action, -1, 1)[0]
          done = self.current_step &gt;= self.max_steps
          return np.array([self.state, self.current_step / self.max_steps], dtype=np.float32), reward, done, False, {}
  
  # Set up the environment
  env = make_vec_env(SimpleOptEnv, n_envs=4)
  
  # Set up logging
  log_dir = os.path.join(dir, 'simple_opt_tensorboard')
  os.makedirs(log_dir, exist_ok=True)
  
  # Define the learning rate schedule
  def linear_schedule(initial_value: float):
      def func(progress_remaining: float) -&gt; float:
          return progress_remaining * initial_value
      return func
  
  # Create the PPO agent with improved hyperparameters
  model = PPO(&quot;MlpPolicy&quot;, env, 
              learning_rate=1e-3,
              n_steps=512,
              batch_size=64,
              n_epochs=20,
              gamma=0.99,
              gae_lambda=0.95,
              clip_range=0.1,
              ent_coef=0,
              vf_coef=0.5,
              max_grad_norm=0.5,
              verbose=0, 
              tensorboard_log=log_dir)
  
  # Set up evaluation callback
  eval_env = make_vec_env(SimpleOptEnv, n_envs=1)
  eval_callback = EvalCallback(eval_env, eval_freq=1000, verbose=0,
                               deterministic=True, render=False, n_eval_episodes=1)
  
  
  # Train the agent
  print(f'TensorBoard command: tensorboard serve --logdir=&quot;{log_dir}&quot;')
  model.learn(total_timesteps=100000, callback=[eval_callback], tb_log_name='PPO')
  print(&quot;Training completed. You can now run TensorBoard to view the results.&quot;)
  
  
  # Evaluate the trained agent
  mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10)
  print(f&quot;Mean reward: {mean_reward:.2f} +/- {std_reward:.2f}&quot;)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-03T14:08:35.723Z
Comments.Text: Hi @Hank Chen and welcome to AI Stack Exchange! What is the value of the entropy  coefficient hyperparameter in your PPO implementation? Sometimes this coefficient is represented as $\beta$.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-03T16:37:29.403Z
Comments.Text: Hi @DeepQZero! Thanks for your response. I updated my question to include a script that can replicate the issue. I currently use ent_coef=0, since the optimal policy is non-random. I tried values like 0.01 and it does not move the needle much.
Comments.UserDisplayName: Hank Chen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-03T18:01:42.963Z
Comments.Text: My initial thought was that the entropy coefficient was too high - obviously it isn't at 0. I might take a look at this question in my spare time, but I hope someone else has an answer for you soon.
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-05T05:11:05.917Z
Comments.Text: Thanks! What is really puzzling is that the same hyper-parameter can solve cases in which optimal policy is not near the boundary, and std drops to near zero. But it just cannot really converge near the boundary. I wander if some features of PPO, or maybe it's the use of clipping to deal with unbounded actions causes this...
Comments.UserDisplayName: Hank Chen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-07T06:29:30.467Z
Comments.Text: I am guessing that it's the randomness of your first reward that the value function has a hard time predicting. To check try to either computer the reward after setting the new self.state which would make the reward completely deterministic, alternatively I would decrease gamma to maybe 0.95 to disregard long-term reward structure and focus more on immediate rewards.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-07T20:17:16.067Z
Comments.Text: Hi @Chillston! Thanks for pointing this out. I realized the same issue this morning so I changed the code a bit to just set first x to be the theoretical optimal value. It does make the learning process a lot less noisy, but the the same problem persist: if the goal is to optimize -(x+0.1)^2 + 1 over [-1,1], the agent can find the optimal policy (x=-0.1) each period. But if reward is -(x+0.9)^2 + 1, it will only find x=-1. It seems that when solution is too near the boundary, std cannot effectively go down, because too many sampled values are clipped, but I am not sure how to fix this..
Comments.UserDisplayName: Hank Chen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-09T07:22:01.793Z
Comments.Text: Ah that's not what I meant, sorry - I mean to swap the two lines of code that computes the reward and that sets the new state otherwise your reward would be lagging behind one action, no?
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-09T07:23:48.990Z
Comments.Text: Also more things to consider: 1) If your assumption is true, you would see it in the entropy of the action choices. Does that stay large over training? 2) Your observations contain time but the action doesn't depend on time so this might make it much harder to learn the correct action 3) I would still decrease `gamma` (and `gae_lambda`) as your action also doesn't depend on history 4) You have a large number of epochs, this can also lead to premature convergence of value/policy function. 5) Check your value function loss too, if large increase `vf_coef`.
  It's a lot but hope it helps :)
Comments.UserDisplayName: Chillston
Tags: <reinforcement-learning><proximal-policy-optimization><stable-baselines>
Title: PPO has hard time finding solutions at the boundary of action space
UserDisplayName: Hank Chen
ThreadText: # PPO has hard time finding solutions at the boundary of action space
  
  (Asked by: username_0 on 2024-10-03)
  
  I am applying PPO to a custom environment and it struggles when the optimal action seems to be at the boundary of the action space. I replicated it in the following simple environment. There are two state variables: time and a variable x, and at each step the reward is -(x+a)^2 + 1, where a is a parameter. At each step the agent should choose x for the next period. x must be in a certain interval. Totally run 10 periods.
  The theoretical optimal policy should be just to choose -a at each step. But what I have found is that when the optimal solution is close to the boundary, for example when period reward is -(x+0.9)^2 + 1 and the constraint for x is [-1,1], PPO at best can give me a policy that picks -1 at each period. PPO can solve the problem though, when the solution is "interior" enough (e.g., with reward -x^2 + 1), and it converges to a very deterministic policy.
  I looked into what is going on during the training, and it seems that when solution is near the boundary, the policy never goes down to a very deterministic one (std of the Gaussian policy remains at a relatively high level, ~0.3-0.4), and hence it just cannot settle down. In fact, mean_reward does not fully converge (as it should have, in such a simple case).
  I can kind of see why this happens, but I do not fully understand what feature of PPO lead to this, and how to fix it. I am also surprised that the optimal solution is not that near the boundary yet my PPO agent cannot find it. Is this something that can be fixed via hyperparameter tuning (if so, how?), or is there any other algorithm better for this kind of tasks (I wanted to stick to PPO because it works for the more complicated task that I am actually trying to solve)?
  I am using stable-baseline 3. Here is a minimum script that can re-produce the problem
  import numpy as np
  import gymnasium as gym
  from gymnasium import spaces
  from stable_baselines3 import PPO
  from stable_baselines3.common.evaluation import evaluate_policy
  from stable_baselines3.common.callbacks import EvalCallback
  from stable_baselines3.common.utils import set_random_seed
  from stable_baselines3.common.env_util import make_vec_env
  import os
  import torch
  
  # Setting up the environment
  home_dir = os.path.expanduser("~")
  dir = home_dir+''
  GLOBAL_SEED = 0
  set_random_seed(GLOBAL_SEED) # This is to ensure reproducibility
  #%%
  class SimpleOptEnv(gym.Env):
      def __init__(self, max_steps=10):
          super(SimpleOptEnv, self).__init__()
          self.action_space = spaces.Box(low=-1, high=1, shape=(1,), dtype=np.float32)
          self.observation_space = spaces.Box(low=-1, high=1, shape=(2,), dtype=np.float32)
          self.max_steps = max_steps
          self.current_step = 0
          self.state = None
  
      def reset(self, seed=GLOBAL_SEED, options=None):
          super().reset(seed=seed)
          self.current_step = 0
          self.state = -0.9
          return np.array([self.state, self.current_step / self.max_steps], dtype=np.float32), {}
  
      def step(self, action):
          self.current_step += 1
          reward = -1 * (self.state+0.9)**2 + 1
          self.state = np.clip(action, -1, 1)[0]
          done = self.current_step >= self.max_steps
          return np.array([self.state, self.current_step / self.max_steps], dtype=np.float32), reward, done, False, {}
  
  # Set up the environment
  env = make_vec_env(SimpleOptEnv, n_envs=4)
  
  # Set up logging
  log_dir = os.path.join(dir, 'simple_opt_tensorboard')
  os.makedirs(log_dir, exist_ok=True)
  
  # Define the learning rate schedule
  def linear_schedule(initial_value: float):
      def func(progress_remaining: float) -> float:
          return progress_remaining * initial_value
      return func
  
  # Create the PPO agent with improved hyperparameters
  model = PPO("MlpPolicy", env, 
              learning_rate=1e-3,
              n_steps=512,
              batch_size=64,
              n_epochs=20,
              gamma=0.99,
              gae_lambda=0.95,
              clip_range=0.1,
              ent_coef=0,
              vf_coef=0.5,
              max_grad_norm=0.5,
              verbose=0, 
              tensorboard_log=log_dir)
  
  # Set up evaluation callback
  eval_env = make_vec_env(SimpleOptEnv, n_envs=1)
  eval_callback = EvalCallback(eval_env, eval_freq=1000, verbose=0,
                               deterministic=True, render=False, n_eval_episodes=1)
  
  
  # Train the agent
  print(f'TensorBoard command: tensorboard serve --logdir="{log_dir}"')
  model.learn(total_timesteps=100000, callback=[eval_callback], tb_log_name='PPO')
  print("Training completed. You can now run TensorBoard to view the results.")
  
  
  # Evaluate the trained agent
  mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10)
  print(f"Mean reward: {mean_reward:.2f} +/- {std_reward:.2f}")
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, stable-baselines
  > **Comments:**
  > **username_1**: Hi @Hank Chen and welcome to AI Stack Exchange! What is the value of the entropy  coefficient hyperparameter in your PPO implementation? Sometimes this coefficient is represented as $\beta$.  
  > **username_0**: Hi @DeepQZero! Thanks for your response. I updated my question to include a script that can replicate the issue. I currently use ent_coef=0, since the optimal policy is non-random. I tried values like 0.01 and it does not move the needle much.  
  > **username_1**: My initial thought was that the entropy coefficient was too high - obviously it isn't at 0. I might take a look at this question in my spare time, but I hope someone else has an answer for you soon.  
  > **username_0**: Thanks! What is really puzzling is that the same hyper-parameter can solve cases in which optimal policy is not near the boundary, and std drops to near zero. But it just cannot really converge near the boundary. I wander if some features of PPO, or maybe it's the use of clipping to deal with unbounded actions causes this...  
  > **username_2**: I am guessing that it's the randomness of your first reward that the value function has a hard time predicting. To check try to either computer the reward after setting the new self.state which would make the reward completely deterministic, alternatively I would decrease gamma to maybe 0.95 to disregard long-term reward structure and focus more on immediate rewards.  
  > **username_0**: Hi @Chillston! Thanks for pointing this out. I realized the same issue this morning so I changed the code a bit to just set first x to be the theoretical optimal value. It does make the learning process a lot less noisy, but the the same problem persist: if the goal is to optimize -(x+0.1)^2 + 1 over [-1,1], the agent can find the optimal policy (x=-0.1) each period. But if reward is -(x+0.9)^2 + 1, it will only find x=-1. It seems that when solution is too near the boundary, std cannot effectively go down, because too many sampled values are clipped, but I am not sure how to fix this..  
  > **username_2**: Ah that's not what I meant, sorry - I mean to swap the two lines of code that computes the reward and that sets the new state otherwise your reward would be lagging behind one action, no?  
  > **username_2**: Also more things to consider: 1) If your assumption is true, you would see it in the entropy of the action choices. Does that stay large over training? 2) Your observations contain time but the action doesn't depend on time so this might make it much harder to learn the correct action 3) I would still decrease `gamma` (and `gae_lambda`) as your action also doesn't depend on history 4) You have a large number of epochs, this can also lead to premature convergence of value/policy function. 5) Check your value function loss too, if large increase `vf_coef`. It's a lot but hope it helps :)  
  
  ---
  
  ## 0 answers
  
  