Answers.Body: <p>The algorithm can be summarized by the following equation, as done in this <a href="https://danieltakeshi.github.io/2020/06/28/offline-rl/" rel="nofollow noreferrer">post</a>:</p>
  <p><span class="math-container">$$\mathcal{L}(\theta) = \ell_k \left( r + \gamma . \left(max_{a'~s.t. \frac{G_\omega(a'|s')}{max~\hat{a}~G_\omega(\hat{a}|s')} &gt; \tau}{Q_{\theta'}}(s', a') \right) - Q_{\theta}(s, a) \right)$$</span></p>
  <p>Let's imagine that we have the following <span class="math-container">$Q_{\theta}(s, a)$</span> for 3 discrete actions: <code>q = np.array([15.2, 15, 15.4])</code></p>
  <p>In the generative model <span class="math-container">$G_\omega$</span> the output probabilities are: <code>p = np.array([0.2, 0.5, 0.3])</code></p>
  <p>In the first case, let's consider <span class="math-container">$\tau$</span> being 0.3, thus we have:</p>
  <pre><code>p = (p/np.amax(p) &gt; tau)
  max_future_q = np.amax(p * q + (1. - p) * -1e8)
  </code></pre>
  <p>Thus, we have <code>max_future_q</code> equal to 15.4. Depending on the values of <span class="math-container">$Q_{\theta}$</span> and <span class="math-container">$p$</span>, we filter the action according to <span class="math-container">$tau$</span> s.t. <span class="math-container">$tau = 0$</span> is <span class="math-container">$Q_{\theta}$</span> and <span class="math-container">$tau = 1$</span> is behavioral cloning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-28T08:45:53.897Z
Answers.LastActivityDate: 2022-02-16T13:09:50.720Z
Answers.LastEditDate: 2022-02-16T13:09:50.720Z
Answers.UserDisplayName: HenDoNR
Body: <p>For simplicity, let's consider the discrete version of BCQ where the <a href="https://arxiv.org/abs/1910.01708" rel="nofollow noreferrer">paper</a> and the <a href="https://github.com/wakamori/BCQ/blob/master/discrete_BCQ/discrete_BCQ.py" rel="nofollow noreferrer">code</a> are available. In the line 5 of Algorithm 1 we have the following:</p>
  <p><span class="math-container">$$
  a' = \text{argmax}_{a'|G_{\omega}(a', s')/\text{max}~\hat{a}~G_{\omega}(\hat{a}, s')~&gt;~\tau} Q_{\theta}(s', a')
  $$</span></p>
  <p>I have doubts about the predictive model <span class="math-container">$G_{\omega}$</span>. In behavior cloning, it could be an action obtained using supervised learning mapping states to actions. Instead, in BCQ it looks like an probability regarding the actions available. I'm right? And what is the action <span class="math-container">$\hat{a}$</span>?</p>
  <p>EDIT: As far I understood we compare the probability of the action <span class="math-container">$a'$</span> to <span class="math-container">$\hat{a}$</span>, and if it's above the threshold <span class="math-container">$\tau$</span> we calculate the loss. The question is now how should I proceed if this is not true: should I take the next action Q with the highest value?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-27T22:15:25.887Z
Comments.Text: Can you please put your **specific** question in the title rather than writing "doubts regarding..."?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><dqn><imitation-learning><offline-reinforcement-learning>
Title: Action selection in Batch-Constrained Deep Q-learning (BCQ)
UserDisplayName: HenDoNR
ThreadText: # Action selection in Batch-Constrained Deep Q-learning (BCQ)
  
  (Asked by: username_0 on 2022-01-26)
  
  For simplicity, let's consider the discrete version of BCQ where the [paper](https://arxiv.org/abs/1910.01708) and the [code](https://github.com/wakamori/BCQ/blob/master/discrete_BCQ/discrete_BCQ.py) are available. In the line 5 of Algorithm 1 we have the following:
  
  $$
  a' = \text{argmax}\_{a'|G\_{\omega}(a', s')/\text{max}~\hat{a}~G\_{\omega}(\hat{a}, s')~>~\tau} Q\_{\theta}(s', a')
  $$
  
  I have doubts about the predictive model $G\_{\omega}$. In behavior cloning, it could be an action obtained using supervised learning mapping states to actions. Instead, in BCQ it looks like an probability regarding the actions available. I'm right? And what is the action $\hat{a}$?
  
  EDIT: As far I understood we compare the probability of the action $a'$ to $\hat{a}$, and if it's above the threshold $\tau$ we calculate the loss. The question is now how should I proceed if this is not true: should I take the next action Q with the highest value?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, imitation-learning, offline-reinforcement-learning
  
  > **Comments:**
  > **username_1**: Can you please put your **specific** question in the title rather than writing "doubts regarding..."?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-01-28. Score: 0
  
  The algorithm can be summarized by the following equation, as done in this [post](https://danieltakeshi.github.io/2020/06/28/offline-rl/):
  
  $$\mathcal{L}(\theta) = \ell\_k \left( r + \gamma . \left(max\_{a'~s.t. \frac{G\_\omega(a'|s')}{max~\hat{a}~G\_\omega(\hat{a}|s')} > \tau}{Q\_{\theta'}}(s', a') \right) - Q\_{\theta}(s, a) \right)$$
  
  Let's imagine that we have the following $Q\_{\theta}(s, a)$ for 3 discrete actions: `q = np.array([15.2, 15, 15.4])`
  
  In the generative model $G\_\omega$ the output probabilities are: `p = np.array([0.2, 0.5, 0.3])`
  
  In the first case, let's consider $\tau$ being 0.3, thus we have:
  
  ```
  p = (p/np.amax(p) > tau)
  max_future_q = np.amax(p * q + (1. - p) * -1e8)
  
  ```
  
  Thus, we have `max_future_q` equal to 15.4. Depending on the values of $Q\_{\theta}$ and $p$, we filter the action according to $tau$ s.t. $tau = 0$ is $Q\_{\theta}$ and $tau = 1$ is behavioral cloning.
  
  
  ---
  
  