Answers.Body: <p>Reinforcement Learning is really fun because the agent <em>will find any bug</em> in your implementation and will exploit it.</p>
  <pre><code>&gt;&gt;&gt; take_left(0)
  0
  &gt;&gt;&gt; take_left(1)
  -4
  </code></pre>
  <p>The agent figured out your bug with negative values and exploits negative indexing to get to the target faster.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T13:39:08.967Z
Answers.Text: Thanks a lot for that. I am fairly a bit new to python too (coming from C++). Anyway glad that my code is otherwise fine.
Answers.UserDisplayName: Tyrion 
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T13:34:24.043Z
Answers.LastActivityDate: 2021-05-21T13:34:24.043Z
Answers.UserDisplayName: Kostya
Body: <p>I am a bit new to Reinforcement learning. So, I am extremely sorry if I am asking something obvious. I have written a small piece of code to find the optimal policy for a 5x5 grid problem.</p>
  <ul>
  <li>Scenario 1. The agent is only given two choices (<code>Up</code>, <code>Right</code>). I believe, I am getting an optimal policy.</li>
  <li>Scenario 2. The agent is given four choices (<code>Up</code>, <code>Right</code>, <code>Down</code>, <code>Left</code>). I am getting the wrong answer.</li>
  </ul>
  <p>I have represented actions with numbers:</p>
  <pre><code>0 - Right
  1 - Up
  2 - Down
  3 - Left
  </code></pre>
  <p>When the action <code>Up</code> is chosen, with 0.9 probability it will move up or 0.1 probability move right and vice-versa. When the action Down is chosen, with 0.9 probability it will move down or 0.1 probability move left and vice-versa.</p>
  <p>I did not use any convergence criteria. Instead let it run for sufficient iterations. I have indeed confirmed that my optimal state values and policy is converging but to a wrong number. I am attaching the code below:</p>
  <pre><code>def take_right(state):
      if (state/n &lt; n-1): state = state + n
      return state
  
  def take_up(state):
      if (state%n!=n-1): state = state + 1
      return state
  
  def take_left(state):
      if (state/n &gt; 0): state = state - n
      return state
  
  def take_down(state):
      if (state%n &gt; 0): state = state - 1
      return state
  </code></pre>
  <p>Scenario 1 result:</p>
  <p><a href="https://i.sstatic.net/g1w6B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/g1w6B.png" alt="Given only two choices" /></a></p>
  <p>Scenario 2 result:</p>
  <p><a href="https://i.sstatic.net/9oBAM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9oBAM.png" alt="Given four choices" /></a></p>
  <p>Green has a reward of 100 and Blue has a penalty of 100. Rest of the states have a penalty of 1. Discount factor is chosen as 0.5</p>
  <p>Edit:</p>
  <p>This was really silly question. The problem with my code was more pythonic than RL. Check the comments to get the clue.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-21T13:30:27.180Z
Comments.Text: Welcome to AI Stack Exchange. Could you clarify your state representation? The left arrows would be correct in the second diagram if your logic accidentally allowed wrap-around (so going left from left edge would be optimal if it put the agent on the right hand side). But it is hard to tell whether your action processing has blocked that. The left arrow on the bottom row looks correct to me, better than selecting "up" since it avoids the p(0.1 ) of ending in the penalty location from actually travelling right.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><markov-decision-process><bellman-equations><policy-iteration><finite-markov-decision-process>
Title: Converging to a wrong optimal policy if the agent is given more choices
UserDisplayName: Tyrion 
ThreadText: # Converging to a wrong optimal policy if the agent is given more choices
  
  (Asked by: username_0 on 2021-05-21)
  
  I am a bit new to Reinforcement learning. So, I am extremely sorry if I am asking something obvious. I have written a small piece of code to find the optimal policy for a 5x5 grid problem.
  
  * Scenario 1. The agent is only given two choices (`Up`, `Right`). I believe, I am getting an optimal policy.
  * Scenario 2. The agent is given four choices (`Up`, `Right`, `Down`, `Left`). I am getting the wrong answer.
  
  I have represented actions with numbers:
  
  ```
  0 - Right
  1 - Up
  2 - Down
  3 - Left
  
  ```
  
  When the action `Up` is chosen, with 0.9 probability it will move up or 0.1 probability move right and vice-versa. When the action Down is chosen, with 0.9 probability it will move down or 0.1 probability move left and vice-versa.
  
  I did not use any convergence criteria. Instead let it run for sufficient iterations. I have indeed confirmed that my optimal state values and policy is converging but to a wrong number. I am attaching the code below:
  
  ```
  def take_right(state):
      if (state/n < n-1): state = state + n
      return state
  
  def take_up(state):
      if (state%n!=n-1): state = state + 1
      return state
  
  def take_left(state):
      if (state/n > 0): state = state - n
      return state
  
  def take_down(state):
      if (state%n > 0): state = state - 1
      return state
  
  ```
  
  Scenario 1 result:
  
  [![Given only two choices](https://i.sstatic.net/g1w6B.png)](https://i.sstatic.net/g1w6B.png)
  
  Scenario 2 result:
  
  [![Given four choices](https://i.sstatic.net/9oBAM.png)](https://i.sstatic.net/9oBAM.png)
  
  Green has a reward of 100 and Blue has a penalty of 100. Rest of the states have a penalty of 1. Discount factor is chosen as 0.5
  
  Edit:
  
  This was really silly question. The problem with my code was more pythonic than RL. Check the comments to get the clue.
  
  **Tags:** reinforcement-learning, markov-decision-process, bellman-equations, policy-iteration, finite-markov-decision-process
  
  > **Comments:**
  > **username_1**: Welcome to AI Stack Exchange. Could you clarify your state representation? The left arrows would be correct in the second diagram if your logic accidentally allowed wrap-around (so going left from left edge would be optimal if it put the agent on the right hand side). But it is hard to tell whether your action processing has blocked that. The left arrow on the bottom row looks correct to me, better than selecting "up" since it avoids the p(0.1 ) of ending in the penalty location from actually travelling right.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-05-21. Score: 2 | Accepted answer
  
  Reinforcement Learning is really fun because the agent *will find any bug* in your implementation and will exploit it.
  
  ```
  >>> take_left(0)
  0
  >>> take_left(1)
  -4
  
  ```
  
  The agent figured out your bug with negative values and exploits negative indexing to get to the target faster.
  
  > **Comments:**
  > **username_0**: Thanks a lot for that. I am fairly a bit new to python too (coming from C++). Anyway glad that my code is otherwise fine.  
  
  ---
  
  