Body: <p>I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment.</p>
  <p><strong>Environment</strong>:</p>
  <ul>
  <li>A 4x4 grid world with episode starting at (0,0) and terminal state (3,3)</li>
  <li>Four actions: Left, Up, Right, Down</li>
  <li>The reward of -1 for moving into new state from the previous state to a new state. The reward of 0 when reaching the terminal state. The reward of -2 for bouncing off of the boundary</li>
  <li>Epsilon-greedy scheme for action selection.</li>
  </ul>
  <p>All works fine, and the following are the learning results of the agent.</p>
  <p><a href="https://i.sstatic.net/mw96G.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mw96G.jpg" alt="Mean reward of QL-agent after 50 simulation runs" /></a></p>
  <p>Later I ran a test run of my <strong>TRAINED</strong> QL-agent where I used greedy action selection. All I could see in every episode was that my agent start from (0,0), take right to move to (1,0), and then take left to move back to (0,0) again and this goes on and on and on... I check the Q table and it makes sense because the Q-values for these actions justifies such behaviour. But this is not a practical agent should be doing.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-30T08:15:38.280Z
Comments.Text: I don't think you will get an answer due to the data you have provided. The simple conclusion is that you have a mistake somewhere in the implementation. I would suspect something in how you have stored/interpreted the Q table or re-implemented the environment when moving from training to testing.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-30T13:41:23.533Z
Comments.Text: Agreed with Neil Slater, and I also have another question. The return that you converge to is around -20, which seems very low to me. Shouldn't your agent be able to solve this in like 6 steps, achieving a return of -5?
Comments.UserDisplayName: harwiltz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-30T14:25:54.677Z
Comments.Text: @harwiltz I checked and found it reach min. a score of -9, with a Toral reward of -8. But mostly the number of steps in 10s or 20s for a trained agent.
Comments.UserDisplayName: SJa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-30T17:48:05.923Z
Comments.Text: So is it possible that in your evaluation episodes the agent goes back and forth around 15 times before finally going to the goal?
Comments.UserDisplayName: harwiltz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-30T17:49:19.197Z
Comments.Text: Still, this is very strange behavior, especially since there's a negative reward at each timestep. Sounds like your learning algorithm might not be working correctly, since it shouldn't be too hard for it to converge to an optimal (or at least nearly optimal) policy.
Comments.UserDisplayName: harwiltz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T08:00:27.710Z
Comments.Text: I get your point. I have been looking into why it is doing so. Convergence shouldn't be a big issue because the environment is simple, the agent's logic is simple but I am looking into it. Thank you, the discussion helped in better understanding of RL developmental issues for me.
Comments.UserDisplayName: SJa
Tags: <reinforcement-learning><q-learning><environment><testing>
Title: Strange behavior of Q-learning agent after being trained
UserDisplayName: SJa
ThreadText: # Strange behavior of Q-learning agent after being trained
  
  (Asked by: username_0 on 2020-07-30)
  
  I built a simple X\*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment.
  
  **Environment**:
  
  * A 4x4 grid world with episode starting at (0,0) and terminal state (3,3)
  * Four actions: Left, Up, Right, Down
  * The reward of -1 for moving into new state from the previous state to a new state. The reward of 0 when reaching the terminal state. The reward of -2 for bouncing off of the boundary
  * Epsilon-greedy scheme for action selection.
  
  All works fine, and the following are the learning results of the agent.
  
  [![Mean reward of QL-agent after 50 simulation runs](https://i.sstatic.net/mw96G.jpg)](https://i.sstatic.net/mw96G.jpg)
  
  Later I ran a test run of my **TRAINED** QL-agent where I used greedy action selection. All I could see in every episode was that my agent start from (0,0), take right to move to (1,0), and then take left to move back to (0,0) again and this goes on and on and on... I check the Q table and it makes sense because the Q-values for these actions justifies such behaviour. But this is not a practical agent should be doing.
  
  **Tags:** reinforcement-learning, q-learning, environment, testing
  
  > **Comments:**
  > **username_1**: I don't think you will get an answer due to the data you have provided. The simple conclusion is that you have a mistake somewhere in the implementation. I would suspect something in how you have stored/interpreted the Q table or re-implemented the environment when moving from training to testing.  
  > **username_2**: Agreed with username_1 username_1, and I also have another question. The return that you converge to is around -20, which seems very low to me. Shouldn't your agent be able to solve this in like 6 steps, achieving a return of -5?  
  > **username_0**: @username_2 I checked and found it reach min. a score of -9, with a Toral reward of -8. But mostly the number of steps in 10s or 20s for a trained agent.  
  > **username_2**: So is it possible that in your evaluation episodes the agent goes back and forth around 15 times before finally going to the goal?  
  > **username_2**: Still, this is very strange behavior, especially since there's a negative reward at each timestep. Sounds like your learning algorithm might not be working correctly, since it shouldn't be too hard for it to converge to an optimal (or at least nearly optimal) policy.  
  > **username_0**: I get your point. I have been looking into why it is doing so. Convergence shouldn't be a big issue because the environment is simple, the agent's logic is simple but I am looking into it. Thank you, the discussion helped in better understanding of RL developmental issues for me.  
  
  ---
  
  ## 0 answers
  
  