Answers.Body: <p>From the same page's pseudocode for off-policy Monte Carlo Control for estimating the optimal target policy, in the nested inner loop you have:</p>
  <p>If <span class="math-container">$A_t \neq \pi(S_t)$</span> then exit inner Loop (proceed to next episode), otherwise scale importance sampling ratio <span class="math-container">$W$</span> by <span class="math-container">$\frac{1}{b(A_t|S_t)}$</span>.</p>
  <p>Therefore here samples collected towards the end of episodes where the importance sampling ratio is large, contribute more to the estimation of the expected returns of the target policy. These high-weight samples dominate the update process especially if the behavior policy is more exploratory where <span class="math-container">$b(A_t|S_t)$</span> is very small, leading to learning primarily from the tails of episodes in the sense of return/value updates with greedy actions.</p>
  <p>If nongreedy exploratory actions are common in the early portions of long episodes, the said inner loop would always exit before these early non-greedy actions, causing no feedback or learning about their credit assignment. For instance, if an agent went for a suboptimal long route in a large maze caused by early non-greedy actions, there's no learning to correct this due to the above demonstrated tail learning only nature. The optimal route learning would be very slow on average until some episodes <em>happen to</em> enter the optimal route by the behavior policy's early non-greedy actions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-28T05:36:52.937Z
Answers.LastActivityDate: 2024-03-28T05:36:52.937Z
Answers.UserDisplayName: cinch
Body: <p>Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control):</p>
  <blockquote>
  <p>A potential problem is that this method learns only from the tails of episodes, when
  all of the remaining actions in the episode are greedy. If nongreedy actions are common,
  then learning will be slow, particularly for states appearing in the early portions of
  long episodes.</p>
  </blockquote>
  <p>Could someone please explain clearly the above paragraph?</p>
  
Tags: <reinforcement-learning><sutton-barto><monte-carlo-methods><off-policy-methods>
Title: Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?
UserDisplayName: DSPinfinity
ThreadText: # Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?
  
  (Asked by: username_0 on 2024-03-28)
  
  Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control):
  
  > A potential problem is that this method learns only from the tails of episodes, when
  > all of the remaining actions in the episode are greedy. If nongreedy actions are common,
  > then learning will be slow, particularly for states appearing in the early portions of
  > long episodes.
  
  Could someone please explain clearly the above paragraph?
  
  **Tags:** reinforcement-learning, sutton-barto, monte-carlo-methods, off-policy-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-28. Score: 2 | Accepted answer
  
  From the same page's pseudocode for off-policy Monte Carlo Control for estimating the optimal target policy, in the nested inner loop you have:
  
  If $A\_t \neq \pi(S\_t)$ then exit inner Loop (proceed to next episode), otherwise scale importance sampling ratio $W$ by $\frac{1}{b(A\_t|S\_t)}$.
  
  Therefore here samples collected towards the end of episodes where the importance sampling ratio is large, contribute more to the estimation of the expected returns of the target policy. These high-weight samples dominate the update process especially if the behavior policy is more exploratory where $b(A\_t|S\_t)$ is very small, leading to learning primarily from the tails of episodes in the sense of return/value updates with greedy actions.
  
  If nongreedy exploratory actions are common in the early portions of long episodes, the said inner loop would always exit before these early non-greedy actions, causing no feedback or learning about their credit assignment. For instance, if an agent went for a suboptimal long route in a large maze caused by early non-greedy actions, there's no learning to correct this due to the above demonstrated tail learning only nature. The optimal route learning would be very slow on average until some episodes *happen to* enter the optimal route by the behavior policy's early non-greedy actions.
  
  
  ---
  
  