Answers.Body: <blockquote>
  <p>why do we need the line &quot;!=() then exit inner loop&quot;?</p>
  </blockquote>
  <p>To understand this, you need to go back to the importance sampling ratio:</p>
  <p><span class="math-container">$$\rho_{t:t+n} = \prod_{k=1}^n \frac{\pi(A_{t+k}|S_{t+k})}{b(A_{t+k}|S_{t+k})}$$</span></p>
  <p>NB Here I am using the more general stochastic form for the target policy <span class="math-container">$\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbf{Pr}\{A_t = a | S_t = s \}$</span>. Whilst the pseudocode uses <span class="math-container">$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$</span>. You have to convert between them to see how the weighting factor is really working.</p>
  <p>The weighting factor <span class="math-container">$W$</span> in the pseudocode is in fact <span class="math-container">$\rho_{t:T-1}$</span>, calculated step by step working backwards through the trajectory so that it can be used to weight the return from timestep <span class="math-container">$t$</span> for each <span class="math-container">$(s_t, a_t)$</span> pair.</p>
  <p>An important detail is that if any of the <span class="math-container">$\pi(A_{t+k}|S_{t+k}) = 0$</span> within an episode, then <span class="math-container">$\rho_{t:T-1} = 0$</span> also, and this must then be true for all other timesteps before <span class="math-container">$t$</span>.</p>
  <p>The line:</p>
  <p><span class="math-container">$W \leftarrow W \frac{1}{b(A_t|S_t)}$</span></p>
  <p>assumes that <span class="math-container">$\pi(A_{t}|S_{t}) = 1$</span>. This is true for the greedy policy if, and only if <span class="math-container">$A_t$</span> is the current greedy choice. As the code uses notation for deterministic policy, then that is the same as saying <span class="math-container">$\pi(S_{t}) = A_t$</span> from the pseudocode.</p>
  <p>So, when <span class="math-container">$A_t \neq \pi(S_t)$</span>, <span class="math-container">$W$</span> should be set to <span class="math-container">$0$</span>, and this will logically make <span class="math-container">$W = 0$</span> throughout the rest of the loop, and there will be no updates to any Q values.</p>
  <p>Exiting the loop when <span class="math-container">$A_t \neq \pi(S_t)$</span> saves you from adjusting  <span class="math-container">$W = 0$</span> in the next line conditionally, then following through with calculating all the useless <span class="math-container">$0$</span> updates for earlier time steps.</p>
  <p>Or another way to put it: With the <span class="math-container">$W$</span> update of <span class="math-container">$W \leftarrow W \frac{1}{b(A_t|S_t)}$</span>, the loop may continue backwards to estimate expected returns from earlier timesteps <em>only if</em> the agent chose a non-exploring action compared to the current target policy. If an exploring action was taken, then the <span class="math-container">$W$</span> update from the pseudocode would be wrong. Not only would it be wrong, but determining the correct value - <span class="math-container">$W = 0$</span> - would result in no changes to any estimates for the rest of the loop. Easier just to terminate the loop at that point.</p>
  <p>This is why off-policy Monte Carlo control is said to &quot;only learn from the tails of episodes&quot;. If the agent explored a new action, that <em>is</em> learned from, but only from the point after it took the last exploring action to the end of the episode. Earlier exploring actions are also learned from indirectly, as they can put the agent in a different state that may not be reachable by the current target policy, and may turn out to be interesting.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-02T17:53:55.377Z
Answers.Text: @ Neil Slater, the point I was asking was this: why do we need to check $A_t \neq \pi(S_t)$?
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-02T17:59:05.433Z
Answers.Text: @ Neil Slater, I changed the question photo with pseudocode from SB's book. Could you please update your answer based on this  new photo?  I think the updated answer will be okay for me.
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-03T18:18:03.733Z
Answers.Text: @ Neil Slatter. Thank you so much. You are TOTALLY an expert on RL!
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-02T17:45:21.710Z
Answers.LastActivityDate: 2022-02-03T18:29:33.147Z
Answers.LastEditDate: 2022-02-03T18:29:33.147Z
Answers.UserDisplayName: Neil Slater
Body: <p>Could please some expert on reinforcement learning explain the red-box part in the following off-policy MC control? I mean I did not understand what (and why)
  is done in the step shown as a red-box. I understood all points, except the red box part: why do we need the line  &quot;<span class="math-container">$A_t \neq \pi(S_t)$</span> then exit inner loop&quot;? More specifically, why don't we not exit the inner loop when  <span class="math-container">$A_t= \pi(S_t)$</span>?</p>
  <p><a href="https://i.sstatic.net/Sl7KM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Sl7KM.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-02T20:19:53.710Z
Comments.Text: I suppose to more explicitly get at what is probably the primary point of confusion you could add something like: "Why can we not continue using the remainder of experience generated within this episode for lower time steps $t$?"
  
  (assuming that that is indeed what you're thinking)
Comments.UserDisplayName: Dennis Soemers
Tags: <reinforcement-learning><sutton-barto>
Title: Unclear step in off-policy (every-visit) MC Control: why do we need the line: $A_t \neq \pi(S_t)$ then exit inner loop?
UserDisplayName: user3489173
ThreadText: # Unclear step in off-policy (every-visit) MC Control: why do we need the line: $A_t \neq \pi(S_t)$ then exit inner loop?
  
  (Asked by: username_0 on 2022-02-02)
  
  Could please some expert on reinforcement learning explain the red-box part in the following off-policy MC control? I mean I did not understand what (and why)
  is done in the step shown as a red-box. I understood all points, except the red box part: why do we need the line "$A\_t \neq \pi(S\_t)$ then exit inner loop"? More specifically, why don't we not exit the inner loop when $A\_t= \pi(S\_t)$?
  
  [![enter image description here](https://i.sstatic.net/Sl7KM.png)](https://i.sstatic.net/Sl7KM.png)
  
  **Tags:** reinforcement-learning, sutton-barto
  > **Comments:**
  > **username_1**: I suppose to more explicitly get at what is probably the primary point of confusion you could add something like: "Why can we not continue using the remainder of experience generated within this episode for lower time steps $t$?"  (assuming that that is indeed what you're thinking)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-02. Score: 3 | Accepted answer
  
  > why do we need the line "!=() then exit inner loop"?
  
  To understand this, you need to go back to the importance sampling ratio:
  
  $$\rho\_{t:t+n} = \prod\_{k=1}^n \frac{\pi(A\_{t+k}|S\_{t+k})}{b(A\_{t+k}|S\_{t+k})}$$
  
  NB Here I am using the more general stochastic form for the target policy $\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbf{Pr}\{A\_t = a | S\_t = s \}$. Whilst the pseudocode uses $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$. You have to convert between them to see how the weighting factor is really working.
  
  The weighting factor $W$ in the pseudocode is in fact $\rho\_{t:T-1}$, calculated step by step working backwards through the trajectory so that it can be used to weight the return from timestep $t$ for each $(s\_t, a\_t)$ pair.
  
  An important detail is that if any of the $\pi(A\_{t+k}|S\_{t+k}) = 0$ within an episode, then $\rho\_{t:T-1} = 0$ also, and this must then be true for all other timesteps before $t$.
  
  The line:
  
  $W \leftarrow W \frac{1}{b(A\_t|S\_t)}$
  
  assumes that $\pi(A\_{t}|S\_{t}) = 1$. This is true for the greedy policy if, and only if $A\_t$ is the current greedy choice. As the code uses notation for deterministic policy, then that is the same as saying $\pi(S\_{t}) = A\_t$ from the pseudocode.
  
  So, when $A\_t \neq \pi(S\_t)$, $W$ should be set to $0$, and this will logically make $W = 0$ throughout the rest of the loop, and there will be no updates to any Q values.
  
  Exiting the loop when $A\_t \neq \pi(S\_t)$ saves you from adjusting $W = 0$ in the next line conditionally, then following through with calculating all the useless $0$ updates for earlier time steps.
  
  Or another way to put it: With the $W$ update of $W \leftarrow W \frac{1}{b(A\_t|S\_t)}$, the loop may continue backwards to estimate expected returns from earlier timesteps *only if* the agent chose a non-exploring action compared to the current target policy. If an exploring action was taken, then the $W$ update from the pseudocode would be wrong. Not only would it be wrong, but determining the correct value - $W = 0$ - would result in no changes to any estimates for the rest of the loop. Easier just to terminate the loop at that point.
  
  This is why off-policy Monte Carlo control is said to "only learn from the tails of episodes". If the agent explored a new action, that *is* learned from, but only from the point after it took the last exploring action to the end of the episode. Earlier exploring actions are also learned from indirectly, as they can put the agent in a different state that may not be reachable by the current target policy, and may turn out to be interesting.
  
  > **Comments:**
  > **username_0**: @ Neil Slater, the point I was asking was this: why do we need to check $A_t \neq \pi(S_t)$?  
  > **username_0**: @ Neil Slater, I changed the question photo with pseudocode from SB's book. Could you please update your answer based on this  new photo?  I think the updated answer will be okay for me.  
  > **username_0**: @ Neil Slatter. Thank you so much. You are TOTALLY an expert on RL!  
  
  ---
  
  