Answers.Body: <p>I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial <span class="math-container">$\theta$</span> value to show the convergence properties for different choices of <span class="math-container">$\alpha$</span>. (For example maybe something like <span class="math-container">$\theta_0 = [-3; 0]$</span> so the initial probability of right action is ~0.05 and the initial expected cost is quite large.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T19:25:34.077Z
Answers.Text: Yes, I think you are right. I tried setting the initial theta to $(-3, 0)$ and got a curve much more like Sutton and Barto's. The curves are nearly identical for the three values of $\alpha$, so that's another question mark, but I'll leave that one for now.
Answers.UserDisplayName: LarrySnyder610
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T19:45:53.860Z
Answers.Text: This is exactly right. In a toy example such as the above it is clear what a good starting point could be, but in eg a deep neural network this is not the case.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T12:51:11.583Z
Answers.Text: I guess what was confusing is that the pseudocode for the algorithm suggests initializing $\theta$ to 0, which just happens to be a good starting point for this toy problem, but the plot shows something different and there's no comment about why.
Answers.UserDisplayName: LarrySnyder610
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T18:54:10.757Z
Answers.LastActivityDate: 2022-08-09T18:54:10.757Z
Answers.UserDisplayName: gwtw14
Body: <p>In Sutton and Barto (<a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">PDF</a>, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the &quot;short corridor with switched actions&quot; environment from Example 13.1. The figure looks like this:</p>
  <p><a href="https://i.sstatic.net/pZhnpl.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pZhnpl.jpg" alt="enter image description here" /></a></p>
  <p>My question is, why is the initial performance so poor? I believe it should be close to optimal even without any training.</p>
  <p>My reasoning is this: If we initialize the algorithm with <span class="math-container">$\boldsymbol{\theta} = \boldsymbol{0}$</span>, then the initial policy has</p>
  <p><span class="math-container">$$\pi({\tt right}|s,\boldsymbol{\theta}) = \frac{e^{\theta_1}}{e^{\theta_1} + e^{\theta_2}} = 0.5.$$</span></p>
  <p>And from the figure in Example 13.1 (p. 323) --</p>
  <p><a href="https://i.sstatic.net/mEb0nl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mEb0nl.png" alt="enter image description here" /></a></p>
  <p>-- we know that a policy that goes <span class="math-container">$\tt{right}$</span> with probability 0.5 in every state has an expected value that is very close to that of the optimal policy, around <span class="math-container">$-12$</span> or so. So shouldn't the policies produced by REINFORCE start close to optimal with essentially no training, and then just improve little to none after that?</p>
  <p>My own experiment supports my hunch:</p>
  <p><a href="https://i.sstatic.net/0lf0Ll.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0lf0Ll.png" alt="enter image description here" /></a></p>
  <p>But it's possible I'm doing something wrong, both theoretically and computationally.</p>
  
Tags: <reinforcement-learning><sutton-barto><reinforce>
Title: Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?
UserDisplayName: LarrySnyder610
ThreadText: # Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?
  
  (Asked by: username_0 on 2022-08-08)
  
  In Sutton and Barto ([PDF](http://incompleteideas.net/book/bookdraft2017nov5.pdf), page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this:
  
  [![enter image description here](https://i.sstatic.net/pZhnpl.jpg)](https://i.sstatic.net/pZhnpl.jpg)
  
  My question is, why is the initial performance so poor? I believe it should be close to optimal even without any training.
  
  My reasoning is this: If we initialize the algorithm with $\boldsymbol{\theta} = \boldsymbol{0}$, then the initial policy has
  
  $$\pi({\tt right}|s,\boldsymbol{\theta}) = \frac{e^{\theta\_1}}{e^{\theta\_1} + e^{\theta\_2}} = 0.5.$$
  
  And from the figure in Example 13.1 (p. 323) --
  
  [![enter image description here](https://i.sstatic.net/mEb0nl.png)](https://i.sstatic.net/mEb0nl.png)
  
  -- we know that a policy that goes $\tt{right}$ with probability 0.5 in every state has an expected value that is very close to that of the optimal policy, around $-12$ or so. So shouldn't the policies produced by REINFORCE start close to optimal with essentially no training, and then just improve little to none after that?
  
  My own experiment supports my hunch:
  
  [![enter image description here](https://i.sstatic.net/0lf0Ll.png)](https://i.sstatic.net/0lf0Ll.png)
  
  But it's possible I'm doing something wrong, both theoretically and computationally.
  
  **Tags:** reinforcement-learning, sutton-barto, reinforce
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-09. Score: 4 | Accepted answer
  
  I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial $\theta$ value to show the convergence properties for different choices of $\alpha$. (For example maybe something like $\theta\_0 = [-3; 0]$ so the initial probability of right action is ~0.05 and the initial expected cost is quite large.)
  
  > **Comments:**
  > **username_0**: Yes, I think you are right. I tried setting the initial theta to $(-3, 0)$ and got a curve much more like Sutton and Barto's. The curves are nearly identical for the three values of $\alpha$, so that's another question mark, but I'll leave that one for now.  
  > **username_2**: This is exactly right. In a toy example such as the above it is clear what a good starting point could be, but in eg a deep neural network this is not the case.  
  > **username_0**: I guess what was confusing is that the pseudocode for the algorithm suggests initializing $\theta$ to 0, which just happens to be a good starting point for this toy problem, but the plot shows something different and there's no comment about why.  
  
  ---
  
  