Answers.Body: <p>He's trying to turn the requirement &quot;this policy is <span class="math-container">$\epsilon$</span>-soft&quot; into a feature of the environment. That way every policy in the new environment could be turned into an <span class="math-container">$\epsilon$</span>-soft in the old.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-25T23:37:03.857Z
Answers.Text: But if  we are allowed general policies in the new environment  why do we need this requirement because general policies include $epsilon$-soft policies?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T03:33:32.013Z
Answers.Text: The point of $\epsilon$-soft policies is to encourage exploration (important for finding good policies; I think it can be shown that training an RL agent w/ $\epsilon$-soft policies will be exponentially more likely to converge to the highest-reward policy). The point of making a *new* environment is so he doesn't need to put this requirement on the policy anymore.
Answers.UserDisplayName: programjames
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-25T16:51:14.097Z
Answers.LastActivityDate: 2024-03-25T16:51:14.097Z
Answers.UserDisplayName: programjames
Body: <p>Sutton-Barto, page 102:</p>
  <p>In the second paragraph, we have:</p>
  <blockquote>
  <p>Consider a new environment that is just like the original environment,
  except with the requirement that policies be <span class="math-container">$\epsilon$</span>-soft “moved
  inside” the environment.&quot;</p>
  </blockquote>
  <p>Later in this paragraph, it is written as:</p>
  <blockquote>
  <p>The best one can do in this new environment with general policies is
  the same as the best one could do in the original environment with
  <span class="math-container">$\epsilon$</span>-soft policies.</p>
  </blockquote>
  <p>Is there an inconsistency between the two statements because if we are allowed general policies in the new environment why do we need the requirement of <span class="math-container">$\epsilon$</span>-soft policies “moved inside” the new environment?</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts
UserDisplayName: DSPinfinity
ThreadText: # In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts
  
  (Asked by: username_0 on 2024-03-25)
  
  Sutton-Barto, page 102:
  
  In the second paragraph, we have:
  
  > Consider a new environment that is just like the original environment,
  > except with the requirement that policies be $\epsilon$-soft “moved
  > inside” the environment."
  
  Later in this paragraph, it is written as:
  
  > The best one can do in this new environment with general policies is
  > the same as the best one could do in the original environment with
  > $\epsilon$-soft policies.
  
  Is there an inconsistency between the two statements because if we are allowed general policies in the new environment why do we need the requirement of $\epsilon$-soft policies “moved inside” the new environment?
  
  **Tags:** reinforcement-learning, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-25. Score: 1
  
  He's trying to turn the requirement "this policy is $\epsilon$-soft" into a feature of the environment. That way every policy in the new environment could be turned into an $\epsilon$-soft in the old.
  
  > **Comments:**
  > **username_0**: But if  we are allowed general policies in the new environment  why do we need this requirement because general policies include $epsilon$-soft policies?  
  > **username_1**: The point of $\epsilon$-soft policies is to encourage exploration (important for finding good policies; I think it can be shown that training an RL agent w/ $\epsilon$-soft policies will be exponentially more likely to converge to the highest-reward policy). The point of making a *new* environment is so he doesn't need to put this requirement on the policy anymore.  
  
  ---
  
  