Body: <p>I'm struggling to understand the notation used to represent the features within Example 13.1 (Short corridor with switched actions&quot; in the Sutton and Barto RL book.</p>
  <p>I assume as it is a free pdf it is ok to quote this content here, but if I am wrong, please let me know and I can remove it.</p>
  <blockquote>
  <p>Consider the small corridor gridworld shown inset in the graph below. The reward
  is -1 per step, as usual. In each of the three nonterminal states there are only
  two actions, right and left. These actions have their usual consequences in the
  first and third states (left causes no movement in the first state), but in the
  second state they are reversed, so that right moves to the left and left moves to
  the right. The problem is difficult because all the states appear identical under
  the function approximation. In particular, we define <span class="math-container">$\mathbf{x}(s,right)=[1,0]^T$</span> and
  <span class="math-container">$\mathbf{x}(s,left) = [0,1]^T$</span> , for all s. An action-value method with <span class="math-container">$\epsilon$</span> -greedy action selection
  is forced to choose between just two policies: choosing right with high probability
  <span class="math-container">$1-\epsilon / 2$</span> on all steps or choosing left with the same high probability on all time
  steps. If <span class="math-container">$\epsilon = 0.1$</span>, then these two policies achieve a value (at the start state)
  of less than -44 and -82, respectively, as shown in the graph. A method can
  do significantly better if it can learn a specific probability with which to select
  right. The best probability is about 0.59, which achieves a value of about -11.6.</p>
  </blockquote>
  <p>Screenshot of plot below:
  <a href="https://i.sstatic.net/dX0rd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dX0rd.png" alt="enter image description here" /></a></p>
  <p>In the example it states <span class="math-container">$\mathbf{x}(s,right)=[1,0]^T$</span> and <span class="math-container">$\mathbf{x}(s,left) = [0,1]^T$</span> for all s. Seeing feature vectors with relation to states (and not actions) seemed to make sense to me, e.g. for the polynomial construction:</p>
  <p><span class="math-container">$\mathbf{x}(s)=\left(1, s_1, s_2, s_1 s_2\right)^{\top}$</span></p>
  <p>But I don't recall ever seeing it spelled out like this for features incorporating actions as well. In the polynomial case given above, we have a 1x4 vector. What is the dimension of the feature vector given in Example 13.1? Is it 1x2x2? How can we write it out in terms of <span class="math-container">$x(s,a)$</span> rather than two explicit definitions for each action? <span class="math-container">$x(s,a) = [[1,0],[0,1]]$</span>?</p>
  <p>Also, I'm confused on why this example makes a positive case for using policy gradient over value-based methods. If we just used the action values to determine a policy, wouldn't the agent be able to learn the choosing left from the 2nd state result in a higher action value? In such a case I would expect the value under such a policy would be [-3, -2, -1, 0] for each state.</p>
  <p>Finally, I don't understand the y-axis of the plot. This is the value under policy pi (defined by theta) for some state S. What is S here? A single state? The average across all states?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-30T16:06:08.943Z
Comments.Text: Please don't upload **text** as images, it isn't searchable. There are [tools available](https://or.meta.stackexchange.com/a/372/51) to convert PDF to text and MathJax, please edit to limit the image to the graph and its labels only; the remainder should be quoted text ( > "text". ) and MathJax. Thanks.
Comments.UserDisplayName: Rob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-03T02:16:46.850Z
Comments.Text: Thanks, didn't know about MathJax. Updates made.
Comments.UserDisplayName: topher217
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-03T07:56:17.483Z
Comments.Text: Thanks, I searched for the text and found (an answer?): this: https://towardsdatascience.com/reinforcement-learning-policy-approximation-f5606ad3d909 and http://download.cmutschler.de/lectures/FAU_RL_2021/slides/6-02_Policy_Gradients.pdf - including a link to the canonical paper or lecture avoids another resource (book) which copied the content. --- A link to the full paper is useful, and on stickler sites any copied text **must** appear within quotes, or you get a last warning; not sure about here, but it's fairly standard. Thanks for making the previous changes.
Comments.UserDisplayName: Rob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-04T03:33:59.163Z
Comments.Text: Looks like the towardsdatascience link contains some code that would affirm my assumption about the shape of the feature vector, though I'm not seeing anything new to answer my question about why policy gradient would be useful here over an action-value based method. Finally, my last question about what S is in the plot might be answered by Equation 13.4 in the book where it defines $J(\theta)$ to be relative only to the initial state.
Comments.UserDisplayName: topher217
Tags: <policy-gradients><features><feature-engineering>
Title: Understanding the features given in Example 13.1 of Sutton and Barto
UserDisplayName: topher217
ThreadText: # Understanding the features given in Example 13.1 of Sutton and Barto
  
  (Asked by: username_0 on 2022-09-30)
  
  I'm struggling to understand the notation used to represent the features within Example 13.1 (Short corridor with switched actions" in the Sutton and Barto RL book.
  
  I assume as it is a free pdf it is ok to quote this content here, but if I am wrong, please let me know and I can remove it.
  
  > Consider the small corridor gridworld shown inset in the graph below. The reward
  > is -1 per step, as usual. In each of the three nonterminal states there are only
  > two actions, right and left. These actions have their usual consequences in the
  > first and third states (left causes no movement in the first state), but in the
  > second state they are reversed, so that right moves to the left and left moves to
  > the right. The problem is difficult because all the states appear identical under
  > the function approximation. In particular, we define $\mathbf{x}(s,right)=[1,0]^T$ and
  > $\mathbf{x}(s,left) = [0,1]^T$ , for all s. An action-value method with $\epsilon$ -greedy action selection
  > is forced to choose between just two policies: choosing right with high probability
  > $1-\epsilon / 2$ on all steps or choosing left with the same high probability on all time
  > steps. If $\epsilon = 0.1$, then these two policies achieve a value (at the start state)
  > of less than -44 and -82, respectively, as shown in the graph. A method can
  > do significantly better if it can learn a specific probability with which to select
  > right. The best probability is about 0.59, which achieves a value of about -11.6.
  
  Screenshot of plot below:
  [![enter image description here](https://i.sstatic.net/dX0rd.png)](https://i.sstatic.net/dX0rd.png)
  
  In the example it states $\mathbf{x}(s,right)=[1,0]^T$ and $\mathbf{x}(s,left) = [0,1]^T$ for all s. Seeing feature vectors with relation to states (and not actions) seemed to make sense to me, e.g. for the polynomial construction:
  
  $\mathbf{x}(s)=\left(1, s\_1, s\_2, s\_1 s\_2\right)^{\top}$
  
  But I don't recall ever seeing it spelled out like this for features incorporating actions as well. In the polynomial case given above, we have a 1x4 vector. What is the dimension of the feature vector given in Example 13.1? Is it 1x2x2? How can we write it out in terms of $x(s,a)$ rather than two explicit definitions for each action? $x(s,a) = [[1,0],[0,1]]$?
  
  Also, I'm confused on why this example makes a positive case for using policy gradient over value-based methods. If we just used the action values to determine a policy, wouldn't the agent be able to learn the choosing left from the 2nd state result in a higher action value? In such a case I would expect the value under such a policy would be [-3, -2, -1, 0] for each state.
  
  Finally, I don't understand the y-axis of the plot. This is the value under policy pi (defined by theta) for some state S. What is S here? A single state? The average across all states?
  
  **Tags:** policy-gradients, features, feature-engineering
  
  > **Comments:**
  > **username_1**: Please don't upload **text** as images, it isn't searchable. There are [tools available](https://or.meta.stackexchange.com/a/372/51) to convert PDF to text and MathJax, please edit to limit the image to the graph and its labels only; the remainder should be quoted text ( > "text". ) and MathJax. Thanks.  
  > **username_0**: Thanks, didn't know about MathJax. Updates made.  
  > **username_1**: Thanks, I searched for the text and found (an answer?): this: https://towardsdatascience.com/reinforcement-learning-policy-approximation-f5606ad3d909 and http://download.cmutschler.de/lectures/FAU_RL_2021/slides/6-02_Policy_Gradients.pdf - including a link to the canonical paper or lecture avoids another resource (book) which copied the content. --- A link to the full paper is useful, and on stickler sites any copied text **must** appear within quotes, or you get a last warning; not sure about here, but it's fairly standard. Thanks for making the previous changes.  
  > **username_0**: Looks like the towardsdatascience link contains some code that would affirm my assumption about the shape of the feature vector, though I'm not seeing anything new to answer my question about why policy gradient would be useful here over an action-value based method. Finally, my last question about what S is in the plot might be answered by Equation 13.4 in the book where it defines $J(\theta)$ to be relative only to the initial state.  
  
  ---
  
  ## 0 answers
  
  