<p>He's trying to turn the requirement &quot;this policy is <span class="math-container">$\epsilon$</span>-soft&quot; into a feature of the environment. That way every policy in the new environment could be turned into an <span class="math-container">$\epsilon$</span>-soft in the old.</p>

CC BY-SA 4.0
2024-03-25T23:37:03.857Z
But if  we are allowed general policies in the new environment  why do we need this requirement because general policies include $epsilon$-soft policies?
DSPinfinity
CC BY-SA 4.0
2024-03-26T03:33:32.013Z
The point of $\epsilon$-soft policies is to encourage exploration (important for finding good policies; I think it can be shown that training an RL agent w/ $\epsilon$-soft policies will be exponentially more likely to converge to the highest-reward policy). The point of making a *new* environment is so he doesn't need to put this requirement on the policy anymore.
programjames
CC BY-SA 4.0
2024-03-25T16:51:14.097Z
2024-03-25T16:51:14.097Z
programjames
<p>Sutton-Barto, page 102:</p>
<p>In the second paragraph, we have:</p>
<blockquote>
<p>Consider a new environment that is just like the original environment,
except with the requirement that policies be <span class="math-container">$\epsilon$</span>-soft “moved
inside” the environment.&quot;</p>
</blockquote>
<p>Later in this paragraph, it is written as:</p>
<blockquote>
<p>The best one can do in this new environment with general policies is
the same as the best one could do in the original environment with
<span class="math-container">$\epsilon$</span>-soft policies.</p>
</blockquote>
<p>Is there an inconsistency between the two statements because if we are allowed general policies in the new environment why do we need the requirement of <span class="math-container">$\epsilon$</span>-soft policies “moved inside” the new environment?</p>

<reinforcement-learning><sutton-barto>
In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts
DSPinfinity
# In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts

(Asked by: username_0 on 2024-03-25)

Sutton-Barto, page 102:

In the second paragraph, we have:

> Consider a new environment that is just like the original environment,
> except with the requirement that policies be $\epsilon$-soft “moved
> inside” the environment."

Later in this paragraph, it is written as:

> The best one can do in this new environment with general policies is
> the same as the best one could do in the original environment with
> $\epsilon$-soft policies.

Is there an inconsistency between the two statements because if we are allowed general policies in the new environment why do we need the requirement of $\epsilon$-soft policies “moved inside” the new environment?

**Tags:** reinforcement-learning, sutton-barto


---

## 1 answers

### Answer by username_1 on 2024-03-25. Score: 1

He's trying to turn the requirement "this policy is $\epsilon$-soft" into a feature of the environment. That way every policy in the new environment could be turned into an $\epsilon$-soft in the old.

> **Comments:**
> **username_0**: But if  we are allowed general policies in the new environment  why do we need this requirement because general policies include $epsilon$-soft policies?  
> **username_1**: The point of $\epsilon$-soft policies is to encourage exploration (important for finding good policies; I think it can be shown that training an RL agent w/ $\epsilon$-soft policies will be exponentially more likely to converge to the highest-reward policy). The point of making a *new* environment is so he doesn't need to put this requirement on the policy anymore.  

---

