Answers.Body: <p>The agent has probably learned that it's not worthwhile to explore further, and greedily minimizes cost by taking the least expensive action. I haven't looked at the environment, but one simple test could be to increase the entropy bonus of PPO, and see if the policy explores more (you can keep track of the policy's entropy to see if this is indeed the issue).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T05:51:45.063Z
Answers.Text: You were right. I just modified the line in which I invoke the PPO in this way: `model = PPO("MlpPolicy", env, verbose=1, ent_coef=0.05)`. Now everything works fine
Answers.UserDisplayName: Francesco
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T18:28:48.873Z
Answers.LastActivityDate: 2022-08-10T18:28:48.873Z
Answers.UserDisplayName: mdc
Body: <p>I am trying to implement an intelligent agent that can perform penetration testing within the nasim (<a href="https://github.com/Jjschwartz/NetworkAttackSimulator" rel="nofollow noreferrer">link</a>) environment, a network simulator.</p>
  <p>I would like to try to use parametric mode for actions, and therefore I need an agent that can work with OpenAI gym's MultiDiscrete actions since nasim implements OpenAI gym. I have therefore chosen to use PPO's implementation of Stable Baselines3 (<a href="https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html" rel="nofollow noreferrer">link</a>).
  Everything seems to work when I run the agent on a small scenario (tiny scenario). Still, when I try to run it on a somewhat larger environment (tiny-small scenario) the agent after a few hundred thousand timesteps decides to run only the action that costs the least. Here is the code of the agent:</p>
  <pre><code>import gym
  from stable_baselines3 import PPO
  from stable_baselines3.common.env_util import make_vec_env
  import nasim
  
  if __name__ == &quot;__main__&quot;:
  
      # Build the environment
      env = gym.make(&quot;nasim:TinySmall-v2&quot;)
  
      # Train the model
      model = PPO(&quot;MlpPolicy&quot;, env, verbose=1)
      model.learn(total_timesteps=500000)
  
      # Init the environment
      obs = env.reset()
  
      # Init variables
      total_reward = 0
      number_of_steps = 0
      done = False
      
      # Do the test
      while done != True:
          action, _states = model.predict(obs)
          number_of_steps = number_of_steps + 1
          obs, rewards, done, info = env.step(action)
          print(&quot;Action: &quot; + str(action) + &quot;, reward: &quot; + str(rewards))
          total_reward += rewards
          print(&quot;Total reward: &quot; + str(total_reward))
          print(done)
          print(&quot;Steps: &quot; + str(number_of_steps))
          env.render(mode='readable')
  </code></pre>
  <p>I tried looking for the cause of the problem in the algorithm, including watching this explanation video (<a href="https://www.youtube.com/watch?v=5P7I-xPq8u8" rel="nofollow noreferrer">link</a>)., but I could not identify the cause of the problem.</p>
  <p>Step to reproduce the problem:</p>
  <ol>
  <li>Install stable-baselines3: <code>pip install stable-baselines3</code></li>
  <li>Install version 0.9.1 of nasim (version 0.10 is too recent to be compatible with the PPO implementation of stable-baselines 3): <code>pip install -Iv nasim==0.9.1</code></li>
  <li>Install the last required component: <code>sudo apt install python3-tk</code></li>
  <li>Save the agent code written above in a .py file and run it.</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-06T00:14:55.823Z
Comments.Text: The agent has probably learned that it's not worthwhile to explore further, and greedily minimizes cost by taking the least expensive action. I haven't looked at the environment, but one simple test could be to increase the entropy bonus of PPO, and see if the policy explores more (you can keep track of the policy's entropy to see if this is indeed the issue).
Comments.UserDisplayName: mdc
Tags: <reinforcement-learning><proximal-policy-optimization>
Title: Why does a PPO agent perform only the action that costs the least?
UserDisplayName: Francesco
ThreadText: # Why does a PPO agent perform only the action that costs the least?
  
  (Asked by: username_0 on 2022-08-05)
  
  I am trying to implement an intelligent agent that can perform penetration testing within the nasim ([link](https://github.com/Jjschwartz/NetworkAttackSimulator)) environment, a network simulator.
  
  I would like to try to use parametric mode for actions, and therefore I need an agent that can work with OpenAI gym's MultiDiscrete actions since nasim implements OpenAI gym. I have therefore chosen to use PPO's implementation of Stable Baselines3 ([link](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html)).
  Everything seems to work when I run the agent on a small scenario (tiny scenario). Still, when I try to run it on a somewhat larger environment (tiny-small scenario) the agent after a few hundred thousand timesteps decides to run only the action that costs the least. Here is the code of the agent:
  
  ```
  import gym
  from stable_baselines3 import PPO
  from stable_baselines3.common.env_util import make_vec_env
  import nasim
  
  if __name__ == "__main__":
  
      # Build the environment
      env = gym.make("nasim:TinySmall-v2")
  
      # Train the model
      model = PPO("MlpPolicy", env, verbose=1)
      model.learn(total_timesteps=500000)
  
      # Init the environment
      obs = env.reset()
  
      # Init variables
      total_reward = 0
      number_of_steps = 0
      done = False
      
      # Do the test
      while done != True:
          action, _states = model.predict(obs)
          number_of_steps = number_of_steps + 1
          obs, rewards, done, info = env.step(action)
          print("Action: " + str(action) + ", reward: " + str(rewards))
          total_reward += rewards
          print("Total reward: " + str(total_reward))
          print(done)
          print("Steps: " + str(number_of_steps))
          env.render(mode='readable')
  
  ```
  
  I tried looking for the cause of the problem in the algorithm, including watching this explanation video ([link](https://www.youtube.com/watch?v=5P7I-xPq8u8))., but I could not identify the cause of the problem.
  
  Step to reproduce the problem:
  
  1. Install stable-baselines3: `pip install stable-baselines3`
  2. Install version 0.9.1 of nasim (version 0.10 is too recent to be compatible with the PPO implementation of stable-baselines 3): `pip install -Iv nasim==0.9.1`
  3. Install the last required component: `sudo apt install python3-tk`
  4. Save the agent code written above in a .py file and run it.
  
  **Tags:** reinforcement-learning, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: The agent has probably learned that it's not worthwhile to explore further, and greedily minimizes cost by taking the least expensive action. I haven't looked at the environment, but one simple test could be to increase the entropy bonus of PPO, and see if the policy explores more (you can keep track of the policy's entropy to see if this is indeed the issue).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-10. Score: 1 | Accepted answer
  
  The agent has probably learned that it's not worthwhile to explore further, and greedily minimizes cost by taking the least expensive action. I haven't looked at the environment, but one simple test could be to increase the entropy bonus of PPO, and see if the policy explores more (you can keep track of the policy's entropy to see if this is indeed the issue).
  
  > **Comments:**
  > **username_0**: You were right. I just modified the line in which I invoke the PPO in this way: `model = PPO("MlpPolicy", env, verbose=1, ent_coef=0.05)`. Now everything works fine  
  
  ---
  
  