Answers.Body: <p>Well, the way to know that the agent is actually learning is by looking at its behavior while it performs the task, and by comparing against a known optimal performance.</p>
  
  <p>So, does your agent reaches the goal quickly? Does it step out of the grid frequently? What is the maximum possible sum of rewards / minimum number of steps attainable? Is the agent close to that limit? From your graphic, and if I understood correctly your RL problem, the maximum average reward per step should be close to 1 (depending on the specific environment you are using), so I guess you are not so far from the optimal solution. </p>
  
  <p>Also, probably if you keep training for a longer period, your agent will reach a stable solution that might or might not be optimal. If you keep training after that, your curves surely will look like the ones you found online.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-13T09:02:40.577Z
Answers.Text: Thank you for your answer. Yes, the maximum average reward per episode is 1 and yes, the agent a the end achieve a good average reward. My doubt is that it takes so much time and for more than 5000 episodes (which have length 50 each one) it reach the goal less than 40% of the time. The grid is 10x10 and only the goal gives a reward of +1. I wonder if this problem is caused by the fact that in reality the space of movement is infinite since it can move freely out of the grid taking a -1 of penalty for each step outside. The question is why it takes so many episodes to get a nice reward?
Answers.UserDisplayName: FraMan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-13T13:11:35.160Z
Answers.Text: I also tried for 100k episodes by it is unstable. I noticed that it stay many time stuck just outside the grid and it stay there.  Since its actions are (Up,Down,Left,Right,Stay)
Answers.UserDisplayName: FraMan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-12T23:10:09.217Z
Answers.LastActivityDate: 2020-01-12T23:10:09.217Z
Answers.UserDisplayName: Diego Gomez
Answers.Body: <p>This could mostly be depending on your exploration rate. Consider this exploration threshold:</p>
  <pre><code>exploration_threshold = EPS_END + (EPS_START - EPS_END) * math.exp(-1. * episodes / EPS_DECAY)
  sample = random.random()
  if sample &gt; exploration_threshold:
        # take action using the policy
      else:
        # take action at random
  </code></pre>
  <p>Here the <code>EPS_END=0.9</code>, <code>EPS_START=0.05</code> are starting and ending exploration probabilities. This will force the agent explore the action space with a decaying high probability and gain experiences. After a while it will choose actions from the policy with a high probability, and as the policy converges, it yeilds the maximum expected utility. This will help learn faster and cause the decaying exponential shape of the convergence plot.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-20T17:31:08.100Z
Answers.LastActivityDate: 2022-07-20T17:34:37.133Z
Answers.LastEditDate: 2022-07-20T17:34:37.133Z
Answers.UserDisplayName: malintha
Body: <p>I'm building an RL agent using SARSA and Q-Learning for testing its capabilities. </p>
  
  <p>The environment is a 10x10 grid, where it gets a reward of 1 if he reaches the goal while he takes -1 every time he takes a step out of the grid. So, it can freely move out and every time it takes a step outside of the grid it gets -1. </p>
  
  <p>After tuning the main parameters </p>
  
  <ul>
  <li>alpha_val: 0.25 </li>
  <li>discount: 0.99 </li>
  <li>episode_length: 50 </li>
  <li>eps_val: 0.5 </li>
  </ul>
  
  <p>I get the following plot for 10000 episodes (The plot is sampled every 100 episodes): </p>
  
  <p><a href="https://i.sstatic.net/B6iMK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/B6iMK.png" alt="enter image description here"></a></p>
  
  <p>But when I look at the plots online I see usually plots like this one:</p>
  
  <p><a href="https://i.sstatic.net/5d9pi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5d9pi.png" alt="enter image description here"></a></p>
  
  <p>Since I'm new at RL, I'm asking some comments about my outcome or any type of suggestion if anyone of you think that I'm doing something wrong.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-20T15:31:10.240Z
Comments.Text: You need to optimize your action selection method
Comments.UserDisplayName: School of Electrical and Compu
Tags: <machine-learning><reinforcement-learning>
Title: Why is the average reward plot for my reinforcement learning agent different than the usual plots?
UserDisplayName: FraMan
ThreadText: # Why is the average reward plot for my reinforcement learning agent different than the usual plots?
  
  (Asked by: username_0 on 2020-01-12)
  
  I'm building an RL agent using SARSA and Q-Learning for testing its capabilities.
  
  The environment is a 10x10 grid, where it gets a reward of 1 if he reaches the goal while he takes -1 every time he takes a step out of the grid. So, it can freely move out and every time it takes a step outside of the grid it gets -1.
  
  After tuning the main parameters
  
  * alpha\_val: 0.25
  * discount: 0.99
  * episode\_length: 50
  * eps\_val: 0.5
  
  I get the following plot for 10000 episodes (The plot is sampled every 100 episodes):
  
  [![enter image description here](https://i.sstatic.net/B6iMK.png)](https://i.sstatic.net/B6iMK.png)
  
  But when I look at the plots online I see usually plots like this one:
  
  [![enter image description here](https://i.sstatic.net/5d9pi.png)](https://i.sstatic.net/5d9pi.png)
  
  Since I'm new at RL, I'm asking some comments about my outcome or any type of suggestion if anyone of you think that I'm doing something wrong.
  
  **Tags:** machine-learning, reinforcement-learning
  > **Comments:**
  > **username_1**: You need to optimize your action selection method  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-01-12. Score: 2
  
  Well, the way to know that the agent is actually learning is by looking at its behavior while it performs the task, and by comparing against a known optimal performance.
  
  So, does your agent reaches the goal quickly? Does it step out of the grid frequently? What is the maximum possible sum of rewards / minimum number of steps attainable? Is the agent close to that limit? From your graphic, and if I understood correctly your RL problem, the maximum average reward per step should be close to 1 (depending on the specific environment you are using), so I guess you are not so far from the optimal solution.
  
  Also, probably if you keep training for a longer period, your agent will reach a stable solution that might or might not be optimal. If you keep training after that, your curves surely will look like the ones you found online.
  
  > **Comments:**
  > **username_0**: Thank you for your answer. Yes, the maximum average reward per episode is 1 and yes, the agent a the end achieve a good average reward. My doubt is that it takes so much time and for more than 5000 episodes (which have length 50 each one) it reach the goal less than 40% of the time. The grid is 10x10 and only the goal gives a reward of +1. I wonder if this problem is caused by the fact that in reality the space of movement is infinite since it can move freely out of the grid taking a -1 of penalty for each step outside. The question is why it takes so many episodes to get a nice reward?  
  > **username_0**: I also tried for 100k episodes by it is unstable. I noticed that it stay many time stuck just outside the grid and it stay there.  Since its actions are (Up,Down,Left,Right,Stay)  
  
  ---
  
  ### Answer by username_3 on 2022-07-20. Score: 0
  
  This could mostly be depending on your exploration rate. Consider this exploration threshold:
  exploration_threshold = EPS_END + (EPS_START - EPS_END) * math.exp(-1. * episodes / EPS_DECAY)
  sample = random.random()
  if sample > exploration_threshold:
        # take action using the policy
      else:
        # take action at random
  
  Here the EPS_END=0.9, EPS_START=0.05 are starting and ending exploration probabilities. This will force the agent explore the action space with a decaying high probability and gain experiences. After a while it will choose actions from the policy with a high probability, and as the policy converges, it yeilds the maximum expected utility. This will help learn faster and cause the decaying exponential shape of the convergence plot.
  
  
  ---
  
  