Body: <p>Sutton-Barto, page 134, second paragraph:</p>
  <blockquote>
  <p>In these cliff walking results Expected Sarsa was used on-policy, but in general it
  might use a policy different from the target policy <span class="math-container">$\pi$</span> to generate behavior, in which case
  it becomes an off-policy algorithm.</p>
  </blockquote>
  <p>I am confused here. Sutton-Barto was using <span class="math-container">$\pi$</span> for target policy and <span class="math-container">$b$</span> for the behaviour policy. Why now are they saying &quot;target policy <span class="math-container">$\pi$</span> to generate behavior&quot;?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-02T19:52:17.683Z
Comments.Text: I'm not sure I understand what the doubt is, but here they are saying that you may use a behaviour policy $b$ which is different than $\pi$ (the target policy). In SARSA, you usually have $b = \pi$, but this is expected SARSA. If this answers your question, I'll convert it to a format answer
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-02T21:21:08.137Z
Comments.Text: Just look at the sentence again, you've quoted it correctly and then only pulled half the noun phrase into your question . . . it's not talking about a "target policy $\pi$" - it's talking about a "policy different from the target policy $\pi$"!
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-02T21:25:39.397Z
Comments.Text: In my opinion the wording is bad. It must be like this: " it might use a policy different from the policy to generate behavior, in which case it becomes an off-policy algorithm."
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-02T21:27:20.713Z
Comments.Text: It's normal English phrasing. "A cat different from next door's cat just walked past the window". Your suggestion doesn't make sense, and is ambiguous where S&B is not -  it would be like "A cat different from the cat just walked past the window" . . . unless you've missed something out in your comment above?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-02T22:35:12.263Z
Comments.Text: S&B introduced  $\pi$ for target policy and $b$ for the behaviour policy when discussing off-policy. In my opinion, when there is a discussion related to target and behaviour policy (or on-policy, off-policy), they should stick to this notation.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-03T07:05:08.183Z
Comments.Text: If you want a more formal approach with strict notation and less informal English,  there is an alternative to S&B with a good reputation
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-03T07:06:28.440Z
Comments.Text: I will try to find a link to it for you
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-04T07:49:04.353Z
Comments.Text: Nowdays it seems there are dozens. I think the book I am recalling is this https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf (official free version) But there's also this one: https://faculty.utrgv.edu/weidong.kuang/book/RL.pdf that seems like a work in progress. There's also this which looks promising: https://www.mit.edu/~dimitrib/RLCOURSECOMPLETE.pdf As your issue with S&B is with style and consistency I think you could take brief look at each and decide whether they are more in line with you like to learn
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><sutton-barto>
Title: Sutton-Barto confusing notation for the target and behaviour policy in the expected sarsa
UserDisplayName: DSPinfinity
ThreadText: # Sutton-Barto confusing notation for the target and behaviour policy in the expected sarsa
  
  (Asked by: username_0 on 2024-05-02)
  
  Sutton-Barto, page 134, second paragraph:
  
  > In these cliff walking results Expected Sarsa was used on-policy, but in general it
  > might use a policy different from the target policy $\pi$ to generate behavior, in which case
  > it becomes an off-policy algorithm.
  
  I am confused here. Sutton-Barto was using $\pi$ for target policy and $b$ for the behaviour policy. Why now are they saying "target policy $\pi$ to generate behavior"?
  
  **Tags:** reinforcement-learning, sutton-barto
  
  > **Comments:**
  > **username_1**: I'm not sure I understand what the doubt is, but here they are saying that you may use a behaviour policy $b$ which is different than $\pi$ (the target policy). In SARSA, you usually have $b = \pi$, but this is expected SARSA. If this answers your question, I'll convert it to a format answer  
  > **username_2**: Just look at the sentence again, you've quoted it correctly and then only pulled half the noun phrase into your question . . . it's not talking about a "target policy $\pi$" - it's talking about a "policy different from the target policy $\pi$"!  
  > **username_0**: In my opinion the wording is bad. It must be like this: " it might use a policy different from the policy to generate behavior, in which case it becomes an off-policy algorithm."  
  > **username_2**: It's normal English phrasing. "A cat different from next door's cat just walked past the window". Your suggestion doesn't make sense, and is ambiguous where S&B is not -  it would be like "A cat different from the cat just walked past the window" . . . unless you've missed something out in your comment above?  
  > **username_0**: S&B introduced  $\pi$ for target policy and $b$ for the behaviour policy when discussing off-policy. In my opinion, when there is a discussion related to target and behaviour policy (or on-policy, off-policy), they should stick to this notation.  
  > **username_2**: If you want a more formal approach with strict notation and less informal English,  there is an alternative to S&B with a good reputation  
  > **username_2**: I will try to find a link to it for you  
  > **username_2**: Nowdays it seems there are dozens. I think the book I am recalling is this https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf (official free version) But there's also this one: https://faculty.utrgv.edu/weidong.kuang/book/RL.pdf that seems like a work in progress. There's also this which looks promising: https://www.mit.edu/~dimitrib/RLCOURSECOMPLETE.pdf As your issue with S&B is with style and consistency I think you could take brief look at each and decide whether they are more in line with you like to learn  
  
  ---
  
  ## 0 answers
  
  