Answers.Body: <p>It simply means that you start an episode in any possible state-action combination. Consider a gridworld type environment: an example of starting in a state-action pair would be to start in the top left most square with the action selected as being 'down'. If you do this random initialisation for each episode then in the limit of infinite episodes you will see each possible state-action pair infinitely often, ensuring sufficient exploration.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T10:48:44.280Z
Answers.LastActivityDate: 2022-08-10T10:48:44.280Z
Answers.UserDisplayName: David
Body: <p>In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo:</p>
  <blockquote>
  <p>For policy evaluation to work for action
  values, we must assure continual exploration. One way to do this is by specifying that
  the episodes start in a state–action pair, and that every pair has a nonzero probability of
  being selected as the start. This guarantees that all state–action pairs will be visited an
  infinite number of times in the limit of an infinite number of episodes. We call this the
  assumption of exploring starts.</p>
  </blockquote>
  <p>What does it mean for an episode to <strong>start</strong> in a state-action pair? Examples are welcome</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: What does it mean for an episode to start in a state-action pair?
UserDisplayName: user
ThreadText: # What does it mean for an episode to start in a state-action pair?
  
  (Asked by: username_0 on 2022-08-10)
  
  In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo:
  
  > For policy evaluation to work for action
  > values, we must assure continual exploration. One way to do this is by specifying that
  > the episodes start in a state–action pair, and that every pair has a nonzero probability of
  > being selected as the start. This guarantees that all state–action pairs will be visited an
  > infinite number of times in the limit of an infinite number of episodes. We call this the
  > assumption of exploring starts.
  
  What does it mean for an episode to **start** in a state-action pair? Examples are welcome
  
  **Tags:** reinforcement-learning, sutton-barto
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-10. Score: 2 | Accepted answer
  
  It simply means that you start an episode in any possible state-action combination. Consider a gridworld type environment: an example of starting in a state-action pair would be to start in the top left most square with the action selected as being 'down'. If you do this random initialisation for each episode then in the limit of infinite episodes you will see each possible state-action pair infinitely often, ensuring sufficient exploration.
  
  
  ---
  
  