<blockquote>
<p>So, how can I calculate <span class="math-container">$\mu(a)$</span> when using Thompson Sampling based on dropout?</p>
</blockquote>
<p>The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution.</p>
<p>I don't think this is feasible for practical reasons (the agent will learn so much more slowly due to these calculations, you may as well abandon Thompson Sampling and use epsilon-greedy), and you will have to avoid using importance sampling if you also want to use action-selection techniques where there is no easy way to calculate a distribution.</p>
<p>Many forms of Q-learning do not use importance sampling. These typically just reset eligibility traces if the selected action is different from maximising action.</p>

CC BY-SA 4.0
2018-06-15T11:36:47.503Z
2021-01-05T12:13:32.183Z
2021-01-05T12:13:32.183Z
Neil Slater
<p>In some implementations of off-policy Q-learning, we need to know the action probabilities given by the behavior policy <span class="math-container">$\mu(a)$</span> (e.g., if we want to use importance sampling).</p>
<p>In my case, I am using Deep <a href="https://en.wikipedia.org/wiki/Q-learning" rel="nofollow noreferrer">Q-Learning</a> and selecting actions using <a href="https://en.wikipedia.org/wiki/Thompson_sampling" rel="nofollow noreferrer">Thompson Sampling</a>. I implemented this following the approach in <a href="http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html" rel="nofollow noreferrer">&quot;What My Deep Model Doesn't Know...&quot;</a>: I added dropout to my Q-network and select actions by performing a single stochastic forward pass through the Q-network (i.e., with dropout enabled) and choosing the action with the highest Q-value.</p>
<p>So, how can I calculate <span class="math-container">$\mu(a)$</span> when using Thompson Sampling based on dropout?</p>

<reinforcement-learning><dqn><deep-rl><dropout><thompson-sampling>
How to compute the action probabilities with Thompson sampling in deep Q-learning?
nicolas
# How to compute the action probabilities with Thompson sampling in deep Q-learning?

(Asked by: username_0 on 2018-06-15)

In some implementations of off-policy Q-learning, we need to know the action probabilities given by the behavior policy $\mu(a)$ (e.g., if we want to use importance sampling).

In my case, I am using Deep [Q-Learning](https://en.wikipedia.org/wiki/Q-learning) and selecting actions using [Thompson Sampling](https://en.wikipedia.org/wiki/Thompson_sampling). I implemented this following the approach in ["What My Deep Model Doesn't Know..."](http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html): I added dropout to my Q-network and select actions by performing a single stochastic forward pass through the Q-network (i.e., with dropout enabled) and choosing the action with the highest Q-value.

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?

**Tags:** reinforcement-learning, dqn, deep-rl, dropout, thompson-sampling


---

## 1 answers

### Answer by username_1 on 2018-06-15. Score: 1 | Accepted answer

> So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?

The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution.

I don't think this is feasible for practical reasons (the agent will learn so much more slowly due to these calculations, you may as well abandon Thompson Sampling and use epsilon-greedy), and you will have to avoid using importance sampling if you also want to use action-selection techniques where there is no easy way to calculate a distribution.

Many forms of Q-learning do not use importance sampling. These typically just reset eligibility traces if the selected action is different from maximising action.


---

