Answers.Body: <p>You cannot code an <span class="math-container">$\epsilon$</span>-soft policy directly, because it is not specific enough.</p>
  <p>A policy is <span class="math-container">$\epsilon$</span>-soft provided that there is at least a probability of <span class="math-container">$\frac{\epsilon}{|\mathcal{A}|}$</span> for choosing any action, where <span class="math-container">$\mathcal{A}$</span> is the set of all possible actions.</p>
  <blockquote>
  <p>I know how to code the <span class="math-container">$\epsilon$</span>-greedy.</p>
  </blockquote>
  <p>Then you already know how to code the most commonly-used <span class="math-container">$\epsilon$</span>-soft policy, because an <span class="math-container">$\epsilon$</span>-greedy policy <em>is</em> an <span class="math-container">$\epsilon$</span>-soft policy.</p>
  <blockquote>
  <p>there are inequalities in place of equalities which is the issue for coding the <span class="math-container">$\epsilon$</span>-soft</p>
  </blockquote>
  <p>That is correct. In fact <span class="math-container">$\epsilon$</span>-soft can be thought of as a constraint or test. So you could write some code that tested whether any policy was an <span class="math-container">$\epsilon$</span>-soft policy for any given value of <span class="math-container">$\epsilon$</span>. Or you could write code that determined the value of <span class="math-container">$\epsilon$</span> for any policy.</p>
  <p>Slightly harder would be code that forced a supplied policy to meet constraints of being <span class="math-container">$\epsilon$</span>-soft, because adjusting any low probabilities to be high enough would mean reducing other probabilities, and there are multiple ways you could do that.</p>
  <p>However, a really simple way to make any starting policy <span class="math-container">$\pi$</span> into an <span class="math-container">$\epsilon$</span>-soft variant is to make the policy choice in 2 steps - first step choose between the original policy with probability <span class="math-container">$(1-\epsilon)$</span>, and with probability <span class="math-container">$\epsilon$</span> choose a fixed policy with equal probability for each action. Second step, evaluate whichever policy the first step chose to determine the action.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T17:33:47.180Z
Answers.LastActivityDate: 2021-12-07T15:32:52.667Z
Answers.LastEditDate: 2021-12-07T15:32:52.667Z
Answers.UserDisplayName: Neil Slater
Body: <p>I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an <span class="math-container">$\epsilon$</span>-soft policy.</p>
  <p>Can someone tell me how to code an <span class="math-container">$\epsilon$</span>-soft policy?</p>
  <p>I know how to code the <span class="math-container">$\epsilon$</span>-greedy. In <span class="math-container">$\epsilon$</span>-soft, there are inequalities in place of equalities which is the issue for coding the <span class="math-container">$\epsilon$</span>-soft.</p>
  
Tags: <reinforcement-learning><implementation><monte-carlo-methods><on-policy-methods><epsilon-greedy-policy>
Title: How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?
UserDisplayName: A Q
ThreadText: # How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?
  
  (Asked by: username_0 on 2021-11-15)
  
  I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy.
  
  Can someone tell me how to code an $\epsilon$-soft policy?
  
  I know how to code the $\epsilon$-greedy. In $\epsilon$-soft, there are inequalities in place of equalities which is the issue for coding the $\epsilon$-soft.
  
  **Tags:** reinforcement-learning, implementation, monte-carlo-methods, on-policy-methods, epsilon-greedy-policy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-15. Score: 1 | Accepted answer
  
  You cannot code an $\epsilon$-soft policy directly, because it is not specific enough.
  
  A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for choosing any action, where $\mathcal{A}$ is the set of all possible actions.
  
  > I know how to code the $\epsilon$-greedy.
  
  Then you already know how to code the most commonly-used $\epsilon$-soft policy, because an $\epsilon$-greedy policy *is* an $\epsilon$-soft policy.
  
  > there are inequalities in place of equalities which is the issue for coding the $\epsilon$-soft
  
  That is correct. In fact $\epsilon$-soft can be thought of as a constraint or test. So you could write some code that tested whether any policy was an $\epsilon$-soft policy for any given value of $\epsilon$. Or you could write code that determined the value of $\epsilon$ for any policy.
  
  Slightly harder would be code that forced a supplied policy to meet constraints of being $\epsilon$-soft, because adjusting any low probabilities to be high enough would mean reducing other probabilities, and there are multiple ways you could do that.
  
  However, a really simple way to make any starting policy $\pi$ into an $\epsilon$-soft variant is to make the policy choice in 2 steps - first step choose between the original policy with probability $(1-\epsilon)$, and with probability $\epsilon$ choose a fixed policy with equal probability for each action. Second step, evaluate whichever policy the first step chose to determine the action.
  
  
  ---
  
  