Answers.Body: <blockquote>
    <p>Is there a way to estimate a behavior policy from this dataset so that it can be used in an off-policy learning algorithm?</p>
  </blockquote>
  
  <p>If you have enough examples of <span class="math-container">$(s,a)$</span> pairs for each instance of <span class="math-container">$s$</span> then you can simply estimate</p>
  
  <p><span class="math-container">$$b(a|s) = \frac{N(a,s)}{N(s)}$$</span></p>
  
  <p>Where <span class="math-container">$N$</span> counts the number of instances in your dataset. This might be enough to use off-policy with importance sampling.</p>
  
  <p>Alternatively, you can use an off-policy approach that doesn't need importance sampling. The most straightforward one here would be single-step Q learning. The update step for 1-step Q-learning does not depend on behaviour policy, because:</p>
  
  <ul>
  <li><p>The action value being updated <span class="math-container">$Q(s,a)$</span> already assumes <span class="math-container">$a$</span> is being taken, so you don't need any conditional probability there.</p></li>
  <li><p>The TD target <span class="math-container">$r + \gamma \text{max}_{a'}[Q(s',a')]$</span> does not need to be adjusted for behaviour policy, it works with the target policy directly (implied as <span class="math-container">$\pi(s) = \text{argmax}_{a}[Q(s,a)]$</span>) </p></li>
  </ul>
  
  <p>A 2-step Q learning algorithm would need to adjust for likelihood <span class="math-container">$b(a'|s')$</span> in the TD target <span class="math-container">$\frac{\pi(a'|s')}{b(a'|s')}(r + \gamma r' + \gamma^2\text{max}_{a''}[Q(s'',a'')])$</span> - typically <span class="math-container">$\pi(a'|s')$</span> is either 0 or 1, thus making <span class="math-container">$b(a'|s')$</span> irrelevant some of the time. But you would still prefer to know it for performing updates it you can.</p>
  
  <p>If you are making updates offline and off-policy, then single-step Q learning is probably the simplest approach. It will require more update steps overall to reach convergence, but each one will be simpler.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:24:09.847Z
Answers.LastActivityDate: 2020-04-27T15:56:20.757Z
Answers.LastEditDate: 2020-04-27T15:56:20.757Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>You can simply train a policy from the inputs to predict the actions in your dataset. You can use the cross entropy loss for this, i.e. maximize the the log probability that the policy assigns to the actions in the data set when given the corresponding inputs. This is called behavioral cloning.</p>
  
  <p>The result is an approximation of the behavioral policy that lets you compute probability densities of actions. It is an approximation because the dataset is finite, and even more so when you restrict the learned policy to a class of distributions, e.g. Gaussians.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T03:19:23.590Z
Answers.LastActivityDate: 2020-04-28T03:19:23.590Z
Answers.UserDisplayName: danijar
Answers.Body: <p>If your data look like this <span class="math-container">$(s_{1},a_{1},r_{1},s_{2}),(s_{2},a_{2},r_{2},s_{3}),....,$</span> then this sample drawn from a particular behavior policy. So, you do not need to find the behavior policy just Q-Learning to find the optimal policy while following the behavior policy.</p>
  
  <p>If the MDP is too big then consider applying Deep Q Learning. In both cases, the transition probability they have given has no use. But if you use on-policy learning and you know the dynamics of the system(means transition probabilities), I will recommend you to use dynamic programming(if state-space is not quite large). But for your above problem setting, you can not use dynamic programming, you have only one choice to use off-policy learning. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:30:48.353Z
Answers.Text: The behavior policy is given, so there is no need to find the behavior policy(since this data is drawn from a particular behavior policy).
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:34:18.703Z
Answers.Text: The samples you have drawn is from the behavior policy. Someone gave him access to this data. So now he can just use Q-Learning to find the optimal policy. In case of Q-Learning, you do not need the $b(a|s)$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:38:02.287Z
Answers.Text: The edit makes it better. Might be worth clarifying that you mean single-step Q-learning, although that is the most common variant.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:40:08.837Z
Answers.Text: Yes, this statement "So, you do not need to find the behavior policy" is wrong in some cases. But for Q-Learning you still do not need to know this behavior policy.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:55:08.393Z
Answers.Text: I'm going backwards and forwards on n-step Q learning :-). Actually I stand by my first statement - you need to know behaviour policy for n-step Q learning, because the wieghtings may vary when considering any non-zero update.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T16:07:07.913Z
Answers.Text: I do not mention the n-step Q-Learning(since I do not know the n-step Q-Learning algorithm). I refer to the Q-Learning algorithm(you may say it is one-step Q-Learning), there you do not need to know the behavior policy. But still, if you do not agree with me, feel free to command.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:27:12.667Z
Answers.LastActivityDate: 2020-04-27T15:37:19.660Z
Answers.LastEditDate: 2020-04-27T15:37:19.660Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., <span class="math-container">$p(r,s' \mid s,a)$</span>. </p>
  
  <p>Is there a way to estimate a behavior policy from this dataset so that it can be used in an off-policy learning algorithm?</p>
  
Tags: <reinforcement-learning><policies><off-policy-methods>
Title: How to estimate a behavior policy for off-policy learning based on data?
UserDisplayName: ycenycute
ThreadText: # How to estimate a behavior policy for off-policy learning based on data?
  
  (Asked by: username_0 on 2020-04-27)
  
  I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., $p(r,s' \mid s,a)$.
  
  Is there a way to estimate a behavior policy from this dataset so that it can be used in an off-policy learning algorithm?
  
  **Tags:** reinforcement-learning, policies, off-policy-methods
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-04-27. Score: 0 | Accepted answer
  
  > Is there a way to estimate a behavior policy from this dataset so that it can be used in an off-policy learning algorithm?
  
  If you have enough examples of $(s,a)$ pairs for each instance of $s$ then you can simply estimate
  
  $$b(a|s) = \frac{N(a,s)}{N(s)}$$
  
  Where $N$ counts the number of instances in your dataset. This might be enough to use off-policy with importance sampling.
  
  Alternatively, you can use an off-policy approach that doesn't need importance sampling. The most straightforward one here would be single-step Q learning. The update step for 1-step Q-learning does not depend on behaviour policy, because:
  
  * The action value being updated $Q(s,a)$ already assumes $a$ is being taken, so you don't need any conditional probability there.
  * The TD target $r + \gamma \text{max}\_{a'}[Q(s',a')]$ does not need to be adjusted for behaviour policy, it works with the target policy directly (implied as $\pi(s) = \text{argmax}\_{a}[Q(s,a)]$)
  
  A 2-step Q learning algorithm would need to adjust for likelihood $b(a'|s')$ in the TD target $\frac{\pi(a'|s')}{b(a'|s')}(r + \gamma r' + \gamma^2\text{max}\_{a''}[Q(s'',a'')])$ - typically $\pi(a'|s')$ is either 0 or 1, thus making $b(a'|s')$ irrelevant some of the time. But you would still prefer to know it for performing updates it you can.
  
  If you are making updates offline and off-policy, then single-step Q learning is probably the simplest approach. It will require more update steps overall to reach convergence, but each one will be simpler.
  
  
  ---
  
  ### Answer by username_2 on 2020-04-28. Score: 1
  
  You can simply train a policy from the inputs to predict the actions in your dataset. You can use the cross entropy loss for this, i.e. maximize the the log probability that the policy assigns to the actions in the data set when given the corresponding inputs. This is called behavioral cloning.
  
  The result is an approximation of the behavioral policy that lets you compute probability densities of actions. It is an approximation because the dataset is finite, and even more so when you restrict the learned policy to a class of distributions, e.g. Gaussians.
  
  
  ---
  
  ### Answer by username_3 on 2020-04-27. Score: 0
  
  If your data look like this $(s\_{1},a\_{1},r\_{1},s\_{2}),(s\_{2},a\_{2},r\_{2},s\_{3}),....,$ then this sample drawn from a particular behavior policy. So, you do not need to find the behavior policy just Q-Learning to find the optimal policy while following the behavior policy.
  
  If the MDP is too big then consider applying Deep Q Learning. In both cases, the transition probability they have given has no use. But if you use on-policy learning and you know the dynamics of the system(means transition probabilities), I will recommend you to use dynamic programming(if state-space is not quite large). But for your above problem setting, you can not use dynamic programming, you have only one choice to use off-policy learning.
  
  > **Comments:**
  > **username_3**: The behavior policy is given, so there is no need to find the behavior policy(since this data is drawn from a particular behavior policy).  
  > **username_3**: The samples you have drawn is from the behavior policy. Someone gave him access to this data. So now he can just use Q-Learning to find the optimal policy. In case of Q-Learning, you do not need the $b(a|s)$.  
  > **username_1**: The edit makes it better. Might be worth clarifying that you mean single-step Q-learning, although that is the most common variant.  
  > **username_3**: Yes, this statement "So, you do not need to find the behavior policy" is wrong in some cases. But for Q-Learning you still do not need to know this behavior policy.  
  > **username_1**: I'm going backwards and forwards on n-step Q learning :-). Actually I stand by my first statement - you need to know behaviour policy for n-step Q learning, because the wieghtings may vary when considering any non-zero update.  
  > **username_3**: I do not mention the n-step Q-Learning(since I do not know the n-step Q-Learning algorithm). I refer to the Q-Learning algorithm(you may say it is one-step Q-Learning), there you do not need to know the behavior policy. But still, if you do not agree with me, feel free to command.  
  
  ---
  
  