Answers.Body: <p>Did you check the original paper? See algorithm 3.</p>
  <p><a href="https://arxiv.org/abs/1903.08738" rel="nofollow noreferrer">Batch Policy Learning under Constraints</a></p>
  <p>Here is an implementation I found: <a href="https://github.com/clvoloshin/COBS/blob/master/ope/algos/fqe.py" rel="nofollow noreferrer">https://github.com/clvoloshin/COBS/blob/master/ope/algos/fqe.py</a>
  See their OPE paper for details: <a href="http://arxiv.org/abs/1911.06854" rel="nofollow noreferrer">http://arxiv.org/abs/1911.06854</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-28T05:02:51.247Z
Answers.LastActivityDate: 2023-10-02T04:38:50.750Z
Answers.LastEditDate: 2023-10-02T04:38:50.750Z
Answers.UserDisplayName: King T
Body: <p>I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (<a href="https://arxiv.org/abs/2110.06169" rel="nofollow noreferrer">paper</a>), given a dataset <span class="math-container">$\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$</span> of transitions. I think I have implemented IQL correctly, so now I have a learned policy <span class="math-container">$\pi$</span> that takes in an element of the state space <span class="math-container">$\mathcal S$</span>, and outputs a mean vector and covariance diagonals for a multivariate normal distribution over the action space <span class="math-container">$\mathcal A = \mathbb R^d$</span>. I would like to evaluate this learned policy <span class="math-container">$\pi$</span>.</p>
  <p>This leads me to investigate off-policy evaluation, specifically Fit Q Evaluation (FQE), given a test dataset of transitions <span class="math-container">$\mathcal D_e$</span> as above. I have been reading the offline reinforcement learning paper by Prudencio et al. (<a href="https://arxiv.org/abs/2203.01387" rel="nofollow noreferrer">link</a>). <strong>My problem is that I do not understand how to implement FQE as described in the paper, so I would appreciate guidance for this.</strong> Below I quote the relevant part of the paper and ask some questions.</p>
  <blockquote>
  <p><em><strong>C. Fit Q Evaluation</strong></em></p>
  <p>In fit Q evaluation (FQE), we first train a <span class="math-container">$Q$</span>-function <span class="math-container">$Q_{\phi}^{\pi}$</span> by minimizing the Bellman error under the policy <span class="math-container">$\pi$</span>. Then, we evaluate the policy by computing the average expected return over the states and actions from <span class="math-container">$\mathcal D_e$</span>, such that
  <span class="math-container">\begin{align*}
  \hat J(\pi) = \mathbb E_{\mathbf s, \mathbf a \sim \mathcal D_e} \left[ Q_{\phi}^{\pi}(\mathbf s, \mathbf a) \right].
  \end{align*}</span></p>
  </blockquote>
  <p><strong>Questions:</strong></p>
  <ol>
  <li>Presumably I should use a neural network for <span class="math-container">$Q_{\phi}^{\pi}$</span>, right?</li>
  <li>What is the Bellman error in this context? I have some course notes that give a definition of the Bellman error involving <span class="math-container">$\max_{\mathbf a'}$</span>, which is not workable since my action space <span class="math-container">$\mathcal A$</span> is <span class="math-container">$\mathbb R^d$</span> for some <span class="math-container">$d \in \mathbb Z$</span>.</li>
  </ol>
  <p>I have looked at the off-policy evaluation paper by Voloshin et al. (<a href="https://arxiv.org/abs/1911.06854" rel="nofollow noreferrer">link</a>), which seems to describe FQE in a different way. Specifically,</p>
  <blockquote>
  <p><span class="math-container">$\hat Q(\cdot, \theta) = \lim_{k \to \infty} \hat Q_k$</span>, where
  <span class="math-container">\begin{align*}
  \hat Q_k &amp;= \min_{\theta} \frac{1}{N} \sum_{i = 1}^N \sum_{t = 0}^{\tilde T} \left( \hat Q_{k - 1}(x_t^i, a_t^i ; \theta) - y_t^i \right)^2\\
  y_t^i &amp;\equiv r_t^i + \gamma \mathbb E_{\pi_e} \hat Q_{k - 1}(x_{t + 1}^i, \cdot ; \theta)
  \end{align*}</span></p>
  </blockquote>
  <p>If I understand this correctly, we fit a sequence of neural network <span class="math-container">$Q$</span> functions <span class="math-container">$\hat Q_1, \hat Q_2, \dots$</span> here, trying to minimize the trajectory-average squared distance between <span class="math-container">$\hat Q_{k - 1}(x_t^i, a_t^i ; \theta)$</span> and <span class="math-container">$y_t^i$</span>. But how do I calculate <span class="math-container">$y_t^i$</span> here? I don't know what <span class="math-container">$\mathbb E_{\pi_e} \hat Q_{k - 1}(x_{t + 1}^i, \cdot ; \theta)$</span> means.</p>
  <p>I appreciate any help.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-29T10:07:02.317Z
Comments.Text: Bellman error in general is the MSE error between your estimate of $Q(s, \pi(s))$ and your target, e.g. $r + \gamma Q(s', \pi(s'))$ (keeping notation simple here for how we obtain actions from policy). Your notes probably use a max because it will have been introduced in the context of Q-learning.
Comments.UserDisplayName: David
Tags: <neural-networks><reinforcement-learning><deep-rl><off-policy-methods><offline-reinforcement-learning>
Title: Fit Q Evaluation in offline reinforcement learning
UserDisplayName: Novice
ThreadText: # Fit Q Evaluation in offline reinforcement learning
  
  (Asked by: username_0 on 2023-07-10)
  
  I am working on a PyTorch implementation of Implicit Q-Learning (IQL) ([paper](https://arxiv.org/abs/2110.06169)), given a dataset $\mathcal D = \left\{ (\mathbf s\_i, \mathbf a\_i, \mathbf s\_i', r\_i ) \right\}$ of transitions. I think I have implemented IQL correctly, so now I have a learned policy $\pi$ that takes in an element of the state space $\mathcal S$, and outputs a mean vector and covariance diagonals for a multivariate normal distribution over the action space $\mathcal A = \mathbb R^d$. I would like to evaluate this learned policy $\pi$.
  
  This leads me to investigate off-policy evaluation, specifically Fit Q Evaluation (FQE), given a test dataset of transitions $\mathcal D\_e$ as above. I have been reading the offline reinforcement learning paper by Prudencio et al. ([link](https://arxiv.org/abs/2203.01387)). **My problem is that I do not understand how to implement FQE as described in the paper, so I would appreciate guidance for this.** Below I quote the relevant part of the paper and ask some questions.
  
  > ***C. Fit Q Evaluation***
  >
  > In fit Q evaluation (FQE), we first train a $Q$-function $Q\_{\phi}^{\pi}$ by minimizing the Bellman error under the policy $\pi$. Then, we evaluate the policy by computing the average expected return over the states and actions from $\mathcal D\_e$, such that
  > \begin{align\*}
  > \hat J(\pi) = \mathbb E\_{\mathbf s, \mathbf a \sim \mathcal D\_e} \left[ Q\_{\phi}^{\pi}(\mathbf s, \mathbf a) \right].
  > \end{align\*}
  
  **Questions:**
  
  1. Presumably I should use a neural network for $Q\_{\phi}^{\pi}$, right?
  2. What is the Bellman error in this context? I have some course notes that give a definition of the Bellman error involving $\max\_{\mathbf a'}$, which is not workable since my action space $\mathcal A$ is $\mathbb R^d$ for some $d \in \mathbb Z$.
  
  I have looked at the off-policy evaluation paper by Voloshin et al. ([link](https://arxiv.org/abs/1911.06854)), which seems to describe FQE in a different way. Specifically,
  
  > $\hat Q(\cdot, \theta) = \lim\_{k \to \infty} \hat Q\_k$, where
  > \begin{align\*}
  > \hat Q\_k &= \min\_{\theta} \frac{1}{N} \sum\_{i = 1}^N \sum\_{t = 0}^{\tilde T} \left( \hat Q\_{k - 1}(x\_t^i, a\_t^i ; \theta) - y\_t^i \right)^2\\
  > y\_t^i &\equiv r\_t^i + \gamma \mathbb E\_{\pi\_e} \hat Q\_{k - 1}(x\_{t + 1}^i, \cdot ; \theta)
  > \end{align\*}
  
  If I understand this correctly, we fit a sequence of neural network $Q$ functions $\hat Q\_1, \hat Q\_2, \dots$ here, trying to minimize the trajectory-average squared distance between $\hat Q\_{k - 1}(x\_t^i, a\_t^i ; \theta)$ and $y\_t^i$. But how do I calculate $y\_t^i$ here? I don't know what $\mathbb E\_{\pi\_e} \hat Q\_{k - 1}(x\_{t + 1}^i, \cdot ; \theta)$ means.
  
  I appreciate any help.
  
  **Tags:** neural-networks, reinforcement-learning, deep-rl, off-policy-methods, offline-reinforcement-learning
  
  > **Comments:**
  > **username_1**: Bellman error in general is the MSE error between your estimate of $Q(s, \pi(s))$ and your target, e.g. $r + \gamma Q(s', \pi(s'))$ (keeping notation simple here for how we obtain actions from policy). Your notes probably use a max because it will have been introduced in the context of Q-learning.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-09-28. Score: 0
  
  Did you check the original paper? See algorithm 3.
  
  [Batch Policy Learning under Constraints](https://arxiv.org/abs/1903.08738)
  
  Here is an implementation I found: <https://github.com/clvoloshin/COBS/blob/master/ope/algos/fqe.py>
  See their OPE paper for details: <http://arxiv.org/abs/1911.06854>
  
  
  ---
  
  