Answers.Body: <blockquote>
  <p>So, how can I calculate <span class="math-container">$\mu(a)$</span> when using Thompson Sampling based on dropout?</p>
  </blockquote>
  <p>The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution.</p>
  <p>I don't think this is feasible for practical reasons (the agent will learn so much more slowly due to these calculations, you may as well abandon Thompson Sampling and use epsilon-greedy), and you will have to avoid using importance sampling if you also want to use action-selection techniques where there is no easy way to calculate a distribution.</p>
  <p>Many forms of Q-learning do not use importance sampling. These typically just reset eligibility traces if the selected action is different from maximising action.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-15T11:36:47.503Z
Answers.LastActivityDate: 2021-01-05T12:13:32.183Z
Answers.LastEditDate: 2021-01-05T12:13:32.183Z
Answers.UserDisplayName: Neil Slater
Body: <p>In some implementations of off-policy Q-learning, we need to know the action probabilities given by the behavior policy <span class="math-container">$\mu(a)$</span> (e.g., if we want to use importance sampling).</p>
  <p>In my case, I am using Deep <a href="https://en.wikipedia.org/wiki/Q-learning" rel="nofollow noreferrer">Q-Learning</a> and selecting actions using <a href="https://en.wikipedia.org/wiki/Thompson_sampling" rel="nofollow noreferrer">Thompson Sampling</a>. I implemented this following the approach in <a href="http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html" rel="nofollow noreferrer">&quot;What My Deep Model Doesn't Know...&quot;</a>: I added dropout to my Q-network and select actions by performing a single stochastic forward pass through the Q-network (i.e., with dropout enabled) and choosing the action with the highest Q-value.</p>
  <p>So, how can I calculate <span class="math-container">$\mu(a)$</span> when using Thompson Sampling based on dropout?</p>
  
Tags: <reinforcement-learning><dqn><deep-rl><dropout><thompson-sampling>
Title: How to compute the action probabilities with Thompson sampling in deep Q-learning?
UserDisplayName: nicolas
ThreadText: # How to compute the action probabilities with Thompson sampling in deep Q-learning?
  
  (Asked by: username_0 on 2018-06-15)
  
  In some implementations of off-policy Q-learning, we need to know the action probabilities given by the behavior policy $\mu(a)$ (e.g., if we want to use importance sampling).
  
  In my case, I am using Deep [Q-Learning](https://en.wikipedia.org/wiki/Q-learning) and selecting actions using [Thompson Sampling](https://en.wikipedia.org/wiki/Thompson_sampling). I implemented this following the approach in ["What My Deep Model Doesn't Know..."](http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html): I added dropout to my Q-network and select actions by performing a single stochastic forward pass through the Q-network (i.e., with dropout enabled) and choosing the action with the highest Q-value.
  
  So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, dropout, thompson-sampling
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-06-15. Score: 1 | Accepted answer
  
  > So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?
  
  The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution.
  
  I don't think this is feasible for practical reasons (the agent will learn so much more slowly due to these calculations, you may as well abandon Thompson Sampling and use epsilon-greedy), and you will have to avoid using importance sampling if you also want to use action-selection techniques where there is no easy way to calculate a distribution.
  
  Many forms of Q-learning do not use importance sampling. These typically just reset eligibility traces if the selected action is different from maximising action.
  
  
  ---
  
  