Answers.Body: <p>Say we are in state <span class="math-container">$s$</span>, and <span class="math-container">$b$</span> chooses <span class="math-container">$a'$</span> but <span class="math-container">$\pi$</span> chooses <span class="math-container">$a$</span>. Since <span class="math-container">$\pi$</span> chose <span class="math-container">$a$</span> (<span class="math-container">$\pi(a | s) &gt;0$</span>), coverage implies that <span class="math-container">$b(a|s)&gt;0$</span>. But <span class="math-container">$a'$</span> was ultimately chosen by <span class="math-container">$b$</span>, so it must be that <span class="math-container">$b(a'|s)&gt;0$</span>. Hence there is some chance that <span class="math-container">$a'$</span> is chosen and there is some chance that <span class="math-container">$a$</span> is chosen, meaning <span class="math-container">$b$</span> is stochastic in state <span class="math-container">$s$</span>. Thus the above conclusion follows from the definition of coverage when the policies are not identical under <span class="math-container">$s$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T17:47:31.457Z
Answers.Text: @ sma, perfect answer!
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-06T02:21:04.140Z
Answers.LastActivityDate: 2022-02-07T01:38:12.093Z
Answers.LastEditDate: 2022-02-07T01:38:12.093Z
Answers.UserDisplayName: sma
Body: <p>In <a href="http://incompleteideas.net/book/RLbook2020trimmed.pdf" rel="nofollow noreferrer">Sutton &amp; Barto Reinforcement learning book</a>, page 103 (chapter: off-policy learning
  via importance sampling), the following statement is given:</p>
  <p>&quot;In order to use episodes from <span class="math-container">$b$</span> to estimate values for <span class="math-container">$\pi$</span>, we require that every action
  taken under <span class="math-container">$\pi$</span> is also taken, at least occasionally, under <span class="math-container">$b$</span>. That is, we require that
  <span class="math-container">$\pi$</span>(a|s) &gt; 0 implies <span class="math-container">$b$</span>(a|s) &gt; 0. This is called the assumption of <em>coverage</em>. It follows
  from coverage that <span class="math-container">$b$</span> must be stochastic in states where it is not identical to <span class="math-container">$\pi$</span>. &quot;</p>
  <p>where <span class="math-container">$b$</span> refers to a behavior policy and <span class="math-container">$\pi$</span> to the target optimal policy.
  I don't understand how the above conclusion was obtained from the definition of coverage.</p>
  
Tags: <reinforcement-learning>
Title: Behaviour policy must be stochastic in states where it is not identical to the Optimal policy
UserDisplayName: user3489173
ThreadText: # Behaviour policy must be stochastic in states where it is not identical to the Optimal policy
  
  (Asked by: username_0 on 2022-02-02)
  
  In [Sutton & Barto Reinforcement learning book](http://incompleteideas.net/book/RLbook2020trimmed.pdf), page 103 (chapter: off-policy learning
  via importance sampling), the following statement is given:
  
  "In order to use episodes from $b$ to estimate values for $\pi$, we require that every action
  taken under $\pi$ is also taken, at least occasionally, under $b$. That is, we require that
  $\pi$(a|s) > 0 implies $b$(a|s) > 0. This is called the assumption of *coverage*. It follows
  from coverage that $b$ must be stochastic in states where it is not identical to $\pi$. "
  
  where $b$ refers to a behavior policy and $\pi$ to the target optimal policy.
  I don't understand how the above conclusion was obtained from the definition of coverage.
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-06. Score: 2 | Accepted answer
  
  Say we are in state $s$, and $b$ chooses $a'$ but $\pi$ chooses $a$. Since $\pi$ chose $a$ ($\pi(a | s) >0$), coverage implies that $b(a|s)>0$. But $a'$ was ultimately chosen by $b$, so it must be that $b(a'|s)>0$. Hence there is some chance that $a'$ is chosen and there is some chance that $a$ is chosen, meaning $b$ is stochastic in state $s$. Thus the above conclusion follows from the definition of coverage when the policies are not identical under $s$.
  
  > **Comments:**
  > **username_0**: @ sma, perfect answer!  
  
  ---
  
  