Answers.Body: <p>Because <span class="math-container">$argmin_{m_\tau} E_{x\sim X} \left[L_2^\tau(x - m_\tau)\right]$</span> approximates <span class="math-container">$max_{x \in X} \hspace{0.1cm} x$</span>, then</p>
  <p><span class="math-container">$E_{(s,a,s')\sim \mathcal{D}}\left[\left(max_{a'\in \mathcal{A}\;s.t. \pi_\beta(a'|s')&gt;0} \left(r(s,a) + \gamma Q_{\hat{\theta}}(s',a') - Q_\theta(s, a)\right)\right)^2\right]$</span></p>
  <p>is approximated (with <span class="math-container">$\tau$</span> sufficiently high) by</p>
  <p><span class="math-container">$E_{(s,a,s')\sim \mathcal{D}}\left[argmin_\theta E_{a'\in \mathcal{D}}\left[L_2^\tau\left(r(s,a) + \gamma Q_{\hat{\theta}}(s',a') - Q_\theta(s, a)\right)\right]\right]$</span></p>
  <p>which is then</p>
  <p><span class="math-container">$argmin_\theta E_{(s,a,s',a')\sim \mathcal{D}}\left[L_2^\tau\left(r(s,a) + \gamma Q_{\hat{\theta}}(s',a') - Q_\theta(s, a)\right)\right]$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-06T06:27:38.717Z
Answers.LastActivityDate: 2024-02-06T22:05:12.227Z
Answers.LastEditDate: 2024-02-06T22:05:12.227Z
Answers.UserDisplayName: user118967
Body: <p>I am reading Kostrikov et al.'s &quot;<a href="https://arxiv.org/abs/2110.06169" rel="nofollow noreferrer">Offline Reinforcement Learning with Implicit Q-Learning</a>&quot; but got stuck understanding one particular transformation they use.</p>
  <p>They describe the loss function they would like to minimize:</p>
  <p><a href="https://i.sstatic.net/wnTO0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wnTO0.png" alt="Eq. (4)" /></a></p>
  <p>which is essentially the loss function for Q-learning, with a small twist: because they are interested in offline learning, the actions <span class="math-container">$a'$</span> considered for the maximization are only those produced by the behavior policy <span class="math-container">$\pi_\beta$</span> (essentially, the policy represented by the the offline dataset <span class="math-container">$\mathcal{D}$</span> we are learning from) with probability greater than 0.</p>
  <p>At this point the authors propose to approximate the maximization with <em>expectile regression</em>. They explain expectile regression to be realized by the following minimization of a loss function <span class="math-container">$L_2^\tau$</span> (not replicated here for brevity), where <span class="math-container">$\tau \in [0,1]$</span> is a target expectile:</p>
  <p><a href="https://i.sstatic.net/0m99F.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0m99F.png" alt="enter image description here" /></a></p>
  <p>The intuition is that if <span class="math-container">$\tau$</span> is, say, 0.75, then the minimizing <span class="math-container">$m_\tau$</span> is a point such that <span class="math-container">$P(X &lt; m_\tau) \approx 0.75$</span>. So if you choose a high value of <span class="math-container">$\tau$</span> close to <span class="math-container">$1$</span>, you are in effect computing a maximum value of <span class="math-container">$\mathcal{X}$</span> by stochastic gradient descent, using samples <span class="math-container">$x \sim X$</span>, without necessarily having to iterate over all values of <span class="math-container">$X$</span>.</p>
  <p>Now comes the part that I don't understand. They apply the idea of expectile maximization to Eq. (4) and obtain a new loss function:</p>
  <p><a href="https://i.sstatic.net/4TrHn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4TrHn.png" alt="enter image description here" /></a></p>
  <p>the argument being that minimizing this loss function with respect to <span class="math-container">$\theta$</span> is approximately equivalent to minimizing the loss function in Eq. (4) with respect to <span class="math-container">$\theta$</span>. While I can see the intuition, I am not seeing exactly how this is mathematically justified.</p>
  <p>I think they are using this transformation:</p>
  <p><span class="math-container">$E_{(s,a,s')\sim \mathcal{D}}\left[\left(r(s,a) + \gamma max_{a'\in \mathcal{A}\;s.t. \pi_\beta(a'|s')&gt;0} Q_{\hat{\theta}}(s',a') - Q_\theta(s, a)\right)^2\right]=$</span>
  <span class="math-container">$E_{(s,a,s')\sim \mathcal{D}}\left[\left(max_{a'\in \mathcal{A}\;s.t. \pi_\beta(a'|s')&gt;0} \left(r(s,a) + \gamma Q_{\hat{\theta}}(s',a') - Q_\theta(s, a)\right)\right)^2\right]$</span></p>
  <p>and then, by some step I don't understand, they approximate that by</p>
  <p><span class="math-container">$E_{(s,a,s',a')\sim \mathcal{D}}\left[L_2^\tau\left(r(s,a) + \gamma Q_{\hat{\theta}}(s',a') - Q_\theta(s, a)\right)\right]$</span></p>
  <p>How can that be shown?</p>
  
Tags: <reinforcement-learning><q-learning><offline-reinforcement-learning>
Title: Expectile regression in Implicit Q-Learning
UserDisplayName: user118967
ThreadText: # Expectile regression in Implicit Q-Learning
  
  (Asked by: username_0 on 2024-02-06)
  
  I am reading Kostrikov et al.'s "[Offline Reinforcement Learning with Implicit Q-Learning](https://arxiv.org/abs/2110.06169)" but got stuck understanding one particular transformation they use.
  
  They describe the loss function they would like to minimize:
  
  [![Eq. (4)](https://i.sstatic.net/wnTO0.png)](https://i.sstatic.net/wnTO0.png)
  
  which is essentially the loss function for Q-learning, with a small twist: because they are interested in offline learning, the actions $a'$ considered for the maximization are only those produced by the behavior policy $\pi\_\beta$ (essentially, the policy represented by the the offline dataset $\mathcal{D}$ we are learning from) with probability greater than 0.
  
  At this point the authors propose to approximate the maximization with *expectile regression*. They explain expectile regression to be realized by the following minimization of a loss function $L\_2^\tau$ (not replicated here for brevity), where $\tau \in [0,1]$ is a target expectile:
  
  [![enter image description here](https://i.sstatic.net/0m99F.png)](https://i.sstatic.net/0m99F.png)
  
  The intuition is that if $\tau$ is, say, 0.75, then the minimizing $m\_\tau$ is a point such that $P(X < m\_\tau) \approx 0.75$. So if you choose a high value of $\tau$ close to $1$, you are in effect computing a maximum value of $\mathcal{X}$ by stochastic gradient descent, using samples $x \sim X$, without necessarily having to iterate over all values of $X$.
  
  Now comes the part that I don't understand. They apply the idea of expectile maximization to Eq. (4) and obtain a new loss function:
  
  [![enter image description here](https://i.sstatic.net/4TrHn.png)](https://i.sstatic.net/4TrHn.png)
  
  the argument being that minimizing this loss function with respect to $\theta$ is approximately equivalent to minimizing the loss function in Eq. (4) with respect to $\theta$. While I can see the intuition, I am not seeing exactly how this is mathematically justified.
  
  I think they are using this transformation:
  
  $E\_{(s,a,s')\sim \mathcal{D}}\left[\left(r(s,a) + \gamma max\_{a'\in \mathcal{A}\;s.t. \pi\_\beta(a'|s')>0} Q\_{\hat{\theta}}(s',a') - Q\_\theta(s, a)\right)^2\right]=$
  $E\_{(s,a,s')\sim \mathcal{D}}\left[\left(max\_{a'\in \mathcal{A}\;s.t. \pi\_\beta(a'|s')>0} \left(r(s,a) + \gamma Q\_{\hat{\theta}}(s',a') - Q\_\theta(s, a)\right)\right)^2\right]$
  
  and then, by some step I don't understand, they approximate that by
  
  $E\_{(s,a,s',a')\sim \mathcal{D}}\left[L\_2^\tau\left(r(s,a) + \gamma Q\_{\hat{\theta}}(s',a') - Q\_\theta(s, a)\right)\right]$
  
  How can that be shown?
  
  **Tags:** reinforcement-learning, q-learning, offline-reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-02-06. Score: 2
  
  Because $argmin\_{m\_\tau} E\_{x\sim X} \left[L\_2^\tau(x - m\_\tau)\right]$ approximates $max\_{x \in X} \hspace{0.1cm} x$, then
  
  $E\_{(s,a,s')\sim \mathcal{D}}\left[\left(max\_{a'\in \mathcal{A}\;s.t. \pi\_\beta(a'|s')>0} \left(r(s,a) + \gamma Q\_{\hat{\theta}}(s',a') - Q\_\theta(s, a)\right)\right)^2\right]$
  
  is approximated (with $\tau$ sufficiently high) by
  
  $E\_{(s,a,s')\sim \mathcal{D}}\left[argmin\_\theta E\_{a'\in \mathcal{D}}\left[L\_2^\tau\left(r(s,a) + \gamma Q\_{\hat{\theta}}(s',a') - Q\_\theta(s, a)\right)\right]\right]$
  
  which is then
  
  $argmin\_\theta E\_{(s,a,s',a')\sim \mathcal{D}}\left[L\_2^\tau\left(r(s,a) + \gamma Q\_{\hat{\theta}}(s',a') - Q\_\theta(s, a)\right)\right]$
  
  
  ---
  
  