Answers.Body: <p><strong>Background</strong>: Importance sampling is used in many off-policy RL algorithms when the data is generated with one policy, yet it is being used to update another policy. The policy generating the data is usually called the <em>behavior</em> policy and the policy being updated is the <em>target</em> policy. Since the distribution of data that is generated by the behavior policy is not necessarily the distribution of data that would be generated by the target policy, using the data from the behavior policy might lead to inaccurate estimates when updating the target policy. To mitigate this issue, it is common to introduce the importance sampling ratio
  <span class="math-container">$$\rho_{t:T-1}\ \dot{=}\ \prod_{k=t}^{T-1} \frac{\pi(A_k|S_k)}{b(A_k|S_k)},$$</span>
  where <span class="math-container">$\pi$</span> is the target policy, <span class="math-container">$b$</span> is the behavior policy, <span class="math-container">$t$</span> is the current time step, and <span class="math-container">$T$</span> is the terminal time step. For further questions about this derivation and how the importance sampling ratio can be used to update the target policy, see the discussion near equation 5.3 in <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton &amp; Barto</a>.</p>
  <p><strong>(Case 1) Target Policy is 0:</strong> In the case where one of the terms <span class="math-container">$\pi(A_i|S_i)$</span> is <span class="math-container">$0$</span> in the numerator of the above importance sampling ratio, then the entire product is <span class="math-container">$0$</span>. Intuitively, this sample should have minimal influence in updating the target policy since it has zero probability of being generated by the target policy. As an example, in weighted importance sampling (see equation 5.6), this data sample will have no effect on the estimate.</p>
  <p><strong>(Case 2) Behavior Policy is 0:</strong> In the case where one of the terms <span class="math-container">$b(A_i|S_i)$</span> is <span class="math-container">$0$</span>, the above product would be undefined. Intuitively, if the behavior policy was used to generate the data, then the data must have a nonzero chance of being generated by the behavior policy in the first place; otherwise, we reach a contradiction that results in the undefined product. See Section 5.7 of the same reference for a discussion regarding the design of useful behavior policies to mitigate this and other issues.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T18:02:57.627Z
Answers.LastActivityDate: 2024-06-27T18:09:21.363Z
Answers.LastEditDate: 2024-06-27T18:09:21.363Z
Answers.UserDisplayName: DeepQZero
Answers.Body: <p>Very simply, one of the requirements of off-policy RL to converge, is that the behavioral policy <span class="math-container">$b$</span> has at least the same support of the target policy <span class="math-container">$\pi$</span>, thus:
  <span class="math-container">$$
  \forall s \in S \forall a \in A \,\,\,\,\,\,\,s.t\,\,\,\,\, \pi(a|s)&gt;0\Rightarrow b(a|s)&gt;0
  $$</span></p>
  <p>Thus, you must have a policy that would do at least everything that your knew policy would do</p>
  <p>In the case <span class="math-container">$\pi(a|s) = 0$</span>, then you just don't learn anything</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-16T21:43:59.647Z
Answers.LastActivityDate: 2023-11-16T21:43:59.647Z
Answers.UserDisplayName: Alberto
Body: <p>I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-14T15:39:04.940Z
Comments.Text: Can you be more specific? Can you provide some formulas? You can use mathjax.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><policies><importance-sampling>
Title: What happens when the probability of either one of the policies is 0 in Importance Sampling?
UserDisplayName: A J
ThreadText: # What happens when the probability of either one of the policies is 0 in Importance Sampling?
  
  (Asked by: username_0 on 2023-11-14)
  
  I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?
  
  **Tags:** reinforcement-learning, policies, importance-sampling
  > **Comments:**
  > **username_1**: Can you be more specific? Can you provide some formulas? You can use mathjax.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-11-14. Score: 2
  
  **Background**: Importance sampling is used in many off-policy RL algorithms when the data is generated with one policy, yet it is being used to update another policy. The policy generating the data is usually called the *behavior* policy and the policy being updated is the *target* policy. Since the distribution of data that is generated by the behavior policy is not necessarily the distribution of data that would be generated by the target policy, using the data from the behavior policy might lead to inaccurate estimates when updating the target policy. To mitigate this issue, it is common to introduce the importance sampling ratio
  $$\rho\_{t:T-1}\ \dot{=}\ \prod\_{k=t}^{T-1} \frac{\pi(A\_k|S\_k)}{b(A\_k|S\_k)},$$
  where $\pi$ is the target policy, $b$ is the behavior policy, $t$ is the current time step, and $T$ is the terminal time step. For further questions about this derivation and how the importance sampling ratio can be used to update the target policy, see the discussion near equation 5.3 in [Sutton & Barto](http://incompleteideas.net/book/the-book-2nd.html).
  
  **(Case 1) Target Policy is 0:** In the case where one of the terms $\pi(A\_i|S\_i)$ is $0$ in the numerator of the above importance sampling ratio, then the entire product is $0$. Intuitively, this sample should have minimal influence in updating the target policy since it has zero probability of being generated by the target policy. As an example, in weighted importance sampling (see equation 5.6), this data sample will have no effect on the estimate.
  
  **(Case 2) Behavior Policy is 0:** In the case where one of the terms $b(A\_i|S\_i)$ is $0$, the above product would be undefined. Intuitively, if the behavior policy was used to generate the data, then the data must have a nonzero chance of being generated by the behavior policy in the first place; otherwise, we reach a contradiction that results in the undefined product. See Section 5.7 of the same reference for a discussion regarding the design of useful behavior policies to mitigate this and other issues.
  
  
  ---
  
  ### Answer by username_3 on 2023-11-16. Score: 1
  
  Very simply, one of the requirements of off-policy RL to converge, is that the behavioral policy $b$ has at least the same support of the target policy $\pi$, thus:
  $$
  \forall s \in S \forall a \in A \,\,\,\,\,\,\,s.t\,\,\,\,\, \pi(a|s)>0\Rightarrow b(a|s)>0
  $$
  
  Thus, you must have a policy that would do at least everything that your knew policy would do
  
  In the case $\pi(a|s) = 0$, then you just don't learn anything
  
  
  ---
  
  