Body: <p>This is a question from page 94 of Sutton and Barto's RL book 2020.</p>
  <p>I read in someone's compiled GitHub answers to this book's exercises their answer was: &quot;No because each state in an episode of blackjack is unique.&quot;</p>
  <p>I think my answer is more <em>yes</em>, but I'm thinking in terms of casino blackjack, where they have multiple decks shuffled together and add in the dropped cards back into the deck every X games in order to prevent card counting and 1 game can be seen as an episode. I think in this case that first-visit MC and every-visit MC would have drastically different results, given that, at the start of the new episode, the state of the deck, which is only partially observed, will change the value of taking an action given a state (because I believe the cards left in the deck affect the value of an action, but the deck is not totally observable).</p>
  <p>If this is blackjack, where the discarded cards are added back in and shuffled every episode, I'll agree that it shouldn't make a difference.</p>
  <p>Are there any flaws in my conjecture?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-27T03:05:23.743Z
Comments.Text: I believe the question is here: http://incompleteideas.net/book/RLbook2020.pdf#page=94 The so-called GitHub solution is here: https://github.com/mharbuz/rlbook-exercises/blob/master/chapter5.md Now I have found an arguably more authoritative answer from here https://tams.informatik.uni-hamburg.de/lectures/2015ss/vorlesung/ML/folien/rl2.pdf#page=52 which reads "states never recur in the task, so no difference between first-visit and every-visit MC"
Comments.UserDisplayName: Constructor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T13:40:20.890Z
Comments.Text: Can you please provide the link to the specific Sutton and Barto's book that you're reading (given that there are many versions online)? Also, you say "someone's compiled GitHub answers to this book's exercises". Can you provide the link to the repo?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-03T11:13:46.193Z
Comments.Text: In the book, Sutton and Barto, second edition (Reinforcement Learning: An Introduction) example 5.1 (Blackjack) in the second paragraph the author explicitly states that the version of Blackjack he is considering has in infinite deck with replacement - and so there is no utility to keeping track of the cards dealt.
Comments.UserDisplayName: Quantum Sphinx
Tags: <reinforcement-learning><monte-carlo-methods><sutton-barto>
Title: Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?
UserDisplayName: user8714896
ThreadText: # Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?
  
  (Asked by: username_0 on 2020-12-02)
  
  This is a question from page 94 of Sutton and Barto's RL book 2020.
  
  I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of blackjack is unique."
  
  I think my answer is more *yes*, but I'm thinking in terms of casino blackjack, where they have multiple decks shuffled together and add in the dropped cards back into the deck every X games in order to prevent card counting and 1 game can be seen as an episode. I think in this case that first-visit MC and every-visit MC would have drastically different results, given that, at the start of the new episode, the state of the deck, which is only partially observed, will change the value of taking an action given a state (because I believe the cards left in the deck affect the value of an action, but the deck is not totally observable).
  
  If this is blackjack, where the discarded cards are added back in and shuffled every episode, I'll agree that it shouldn't make a difference.
  
  Are there any flaws in my conjecture?
  
  **Tags:** reinforcement-learning, monte-carlo-methods, sutton-barto
  
  > **Comments:**
  > **username_2**: Can you please provide the link to the specific Sutton and Barto's book that you're reading (given that there are many versions online)? Also, you say "someone's compiled GitHub answers to this book's exercises". Can you provide the link to the repo?  
  > **username_1**: I believe the question is here: http://incompleteideas.net/book/RLbook2020.pdf#page=94 The so-called GitHub solution is here: https://github.com/mharbuz/rlbook-exercises/blob/master/chapter5.md Now I have found an arguably more authoritative answer from here https://tams.informatik.uni-hamburg.de/lectures/2015ss/vorlesung/ML/folien/rl2.pdf#page=52 which reads "states never recur in the task, so no difference between first-visit and every-visit MC"  
  > **username_3**: In the book, Sutton and Barto, second edition (Reinforcement Learning: An Introduction) example 5.1 (Blackjack) in the second paragraph the author explicitly states that the version of Blackjack he is considering has in infinite deck with replacement - and so there is no utility to keeping track of the cards dealt.  
  
  ---
  
  ## 0 answers
  
  