Answers.Body: <p>According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and behaviour policy throughout the trajectory.</p>
  <p>See the calculation below, where the product is happening for all the probabilities throughout the trajectories. (screenshot from Chapter 5, section 5.5 (page 85) of Sutton &amp; Barto)</p>
  <p><a href="https://i.sstatic.net/bNTl3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bNTl3.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T15:25:18.537Z
Answers.Text: I'm not sure whether this formulation from Sutton and Barto assumes that all experiences in the trajectory were generated with a behaviour policy when the value function was fixed (for all those experiences in the trajectory) or not. I think that's the crux of the question. If the value function changes while you're collecting experience for the trajectory, then the behaviour policy, which is derived from the value function, may also change (i.e. take different actions).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-17T20:03:05.540Z
Answers.LastActivityDate: 2020-11-18T06:29:39.553Z
Answers.LastEditDate: 2020-11-18T06:29:39.553Z
Answers.UserDisplayName: Shreyans Jain
Body: <p>When learning off-policy with multi-step returns, we want to update the value of <span class="math-container">$Q(s_1, a_1)$</span> using rewards from the trajectory <span class="math-container">$\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$</span>. We want to learn the target policy <span class="math-container">$\pi$</span> while behaving according to policy <span class="math-container">$\mu$</span>. Therefore, for each transition <span class="math-container">$(s_t, a_t, r_t, s_{t+1})$</span>, we apply the importance ratio <span class="math-container">$\frac{\pi(a_t | s_t)}{\mu(a_t | s_t)}$</span>.</p>
  <p>My question is: if we are training at every step, the behavior policy may change at each step and therefore the transitions of the trajectory <span class="math-container">$\tau$</span> are not obtained from the current behavior policy, but from <span class="math-container">$n$</span> behavior policies. Why do we use the current behavior policy in the importance sampling? Should each transition use the probability of the behavior policy of the timestep at which that transition was collected? For example by storing the likelihood <span class="math-container">$\mu_t(a_t | s_t)$</span> along with the transition?</p>
  
Tags: <reinforcement-learning><off-policy-methods><value-functions><importance-sampling><return>
Title: When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?
UserDisplayName: Federico Taschin
ThreadText: # When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?
  
  (Asked by: username_0 on 2020-11-16)
  
  When learning off-policy with multi-step returns, we want to update the value of $Q(s\_1, a\_1)$ using rewards from the trajectory $\tau = (s\_1, a\_1, r\_1, s\_2, a\_2, r\_2, ..., s\_n, a\_n, r\_n, s\_n+1)$. We want to learn the target policy $\pi$ while behaving according to policy $\mu$. Therefore, for each transition $(s\_t, a\_t, r\_t, s\_{t+1})$, we apply the importance ratio $\frac{\pi(a\_t | s\_t)}{\mu(a\_t | s\_t)}$.
  
  My question is: if we are training at every step, the behavior policy may change at each step and therefore the transitions of the trajectory $\tau$ are not obtained from the current behavior policy, but from $n$ behavior policies. Why do we use the current behavior policy in the importance sampling? Should each transition use the probability of the behavior policy of the timestep at which that transition was collected? For example by storing the likelihood $\mu\_t(a\_t | s\_t)$ along with the transition?
  
  **Tags:** reinforcement-learning, off-policy-methods, value-functions, importance-sampling, return
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-17. Score: 1
  
  According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and behaviour policy throughout the trajectory.
  
  See the calculation below, where the product is happening for all the probabilities throughout the trajectories. (screenshot from Chapter 5, section 5.5 (page 85) of Sutton & Barto)
  
  [![enter image description here](https://i.sstatic.net/bNTl3.png)](https://i.sstatic.net/bNTl3.png)
  
  > **Comments:**
  > **username_2**: I'm not sure whether this formulation from Sutton and Barto assumes that all experiences in the trajectory were generated with a behaviour policy when the value function was fixed (for all those experiences in the trajectory) or not. I think that's the crux of the question. If the value function changes while you're collecting experience for the trajectory, then the behaviour policy, which is derived from the value function, may also change (i.e. take different actions).  
  
  ---
  
  