Answers.Body: <p>No, <span class="math-container">$B\hat{Q}$</span> is not an unbiased estimator. Say that <span class="math-container">$Q(s, a) = 0$</span>, and <span class="math-container">$\hat{Q}(s, a) \sim N(0, 1)$</span>. Then <span class="math-container">$E[max_a \space Q(s, a)] = 0$</span>, but <span class="math-container">$E[max_a \space \hat{Q}(s, a)] &gt; 0$</span> (approximately 1.5387 if you have 10 different actions).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-31T14:38:24.713Z
Answers.LastActivityDate: 2024-03-31T17:11:49.483Z
Answers.LastEditDate: 2024-03-31T17:11:49.483Z
Answers.UserDisplayName: user1502040
Answers.Body: <p>Given your provided definition of unbiased estimator at each input data, and since <span class="math-container">$\hat{Q}$</span> is an unbiased estimator for the true action value <span class="math-container">$Q$</span> at each state-action input pair, applying Bellman backup operator defined in your reference involving a max operator would result in biased <span class="math-container">$B\hat{Q}$</span> not converging to <span class="math-container">$BQ$</span> in expectation at each state-action input pair. The distribution of the maximum of <span class="math-container">$n$</span> independent i.i.d. random variables doesn't necessarily follow the same distribution, it may follow <a href="https://en.wikipedia.org/wiki/Gumbel_distribution" rel="nofollow noreferrer">Gumbel distribution</a> for example, which may not have same expectation as the original distribution, as pointed out by another answer.</p>
  <p>Of course in reality TD off-policy Q-learning cannot ensure there always exists such an unbiased <span class="math-container">$\hat{Q}$</span> estimator at each state-action input.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-31T05:24:30.897Z
Answers.LastActivityDate: 2024-03-31T15:46:15.870Z
Answers.LastEditDate: 2024-03-31T15:46:15.870Z
Answers.UserDisplayName: cinch
Body: <p>This is comes from cs285 2023Fall hw3.
  <a href="https://i.sstatic.net/XqWRS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XqWRS.png" alt="enter image description here" /></a></p>
  <p>In my opinion, if <span class="math-container">$\hat{Q}$</span> is unbiased estimate of <span class="math-container">$Q$</span>, then
  <span class="math-container">$$
  \begin{align}
  \mathbb{E}_{D \sim P}[B_{D}\hat{Q} - B_{D}Q]
  &amp;= \mathbb{E}_{D \sim P}[r(s,a) + \gamma max_{a'}\hat{Q}(s', a') - r(s,a) - \gamma max_{a'}Q(s', a')]\\
  &amp;=\mathbb{E}_{D \sim P}[\gamma max_{a'}\hat{Q}(s', a')- \gamma max_{a'}Q(s', a')]\\
  &amp;= 0
  \end{align}$$</span>
  So <span class="math-container">$\hat{Q}$</span> is unbiased estimate of <span class="math-container">$Q$</span> and the answer is yes.
  Is this right?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T14:39:15.813Z
Comments.Text: Can you provide the link to the homework?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><proofs><temporal-difference-methods><bias>
Title: Is the Bellman backup unbiased?
UserDisplayName: yeebo xie
ThreadText: # Is the Bellman backup unbiased?
  
  (Asked by: username_0 on 2024-03-31)
  
  This is comes from cs285 2023Fall hw3.
  [![enter image description here](https://i.sstatic.net/XqWRS.png)](https://i.sstatic.net/XqWRS.png)
  
  In my opinion, if $\hat{Q}$ is unbiased estimate of $Q$, then
  $$
  \begin{align}
  \mathbb{E}\_{D \sim P}[B\_{D}\hat{Q} - B\_{D}Q]
  &= \mathbb{E}\_{D \sim P}[r(s,a) + \gamma max\_{a'}\hat{Q}(s', a') - r(s,a) - \gamma max\_{a'}Q(s', a')]\\
  &=\mathbb{E}\_{D \sim P}[\gamma max\_{a'}\hat{Q}(s', a')- \gamma max\_{a'}Q(s', a')]\\
  &= 0
  \end{align}$$
  So $\hat{Q}$ is unbiased estimate of $Q$ and the answer is yes.
  Is this right?
  
  **Tags:** reinforcement-learning, proofs, temporal-difference-methods, bias
  
  > **Comments:**
  > **username_1**: Can you provide the link to the homework?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-03-31. Score: 3 | Accepted answer
  
  Given your provided definition of unbiased estimator at each input data, and since $\hat{Q}$ is an unbiased estimator for the true action value $Q$ at each state-action input pair, applying Bellman backup operator defined in your reference involving a max operator would result in biased $B\hat{Q}$ not converging to $BQ$ in expectation at each state-action input pair. The distribution of the maximum of $n$ independent i.i.d. random variables doesn't necessarily follow the same distribution, it may follow [Gumbel distribution](https://en.wikipedia.org/wiki/Gumbel_distribution) for example, which may not have same expectation as the original distribution, as pointed out by another answer.
  
  Of course in reality TD off-policy Q-learning cannot ensure there always exists such an unbiased $\hat{Q}$ estimator at each state-action input.
  
  
  ---
  
  ### Answer by username_2 on 2024-03-31. Score: 4
  
  No, $B\hat{Q}$ is not an unbiased estimator. Say that $Q(s, a) = 0$, and $\hat{Q}(s, a) \sim N(0, 1)$. Then $E[max\_a \space Q(s, a)] = 0$, but $E[max\_a \space \hat{Q}(s, a)] > 0$ (approximately 1.5387 if you have 10 different actions).
  
  
  ---
  
  