Answers.Body: <p>We have for <span class="math-container">$\gamma &lt; 1$</span>:</p>
  <p><span class="math-container">$$ (I-\gamma P^{\pi})^{-1}=I+\gamma P^{\pi} + \gamma^2 (P^{\pi})^2 + \cdots $$</span></p>
  <p>What they mean by <span class="math-container">$(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a) $</span> is the function from <span class="math-container">$\mathcal{S}\times\mathcal{A}\longrightarrow \mathbb{R}$</span>, sending <span class="math-container">$(s,a)\mapsto (I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}$</span>.</p>
  <p>Now, we have</p>
  <p><span class="math-container">$$P^{\pi}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s')}\left[ \frac{\mu(a'\mid s')}{\hat{\pi}_{\beta}(a'\mid s')} \right]. $$</span>
  Similarily,</p>
  <p><span class="math-container">$$(P^{\pi})^2\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}=\mathbb{E}_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s'), s''\sim p(\cdot\mid s',a'), a''\sim \pi(\cdot\mid s'')}\left[ \frac{\mu(a''\mid s'')}{\hat{\pi}_{\beta}(a''\mid s'')} \right]. $$</span></p>
  <p>Adding this together, we have</p>
  <p><span class="math-container">$$(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}_{\beta}(a\mid s)}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t\geq 0} \gamma^t\frac{\mu(a_t\mid s_t)}{\hat{\pi}_{\beta}(a_t\mid s_t)}\bigg\rvert s_0=s, a_0=a \right]. $$</span></p>
  <p>The last quantity is easier, as it does not depend on <span class="math-container">$(s,a)$</span>, so:</p>
  <p><span class="math-container">$$(I-\gamma P^{\pi})^{-1}\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=\sum_{t\geq 0}\gamma^t\frac{C_{r,T,\delta}R_{\max}}{1-\gamma}=C_{r,T,\delta}R_{\max} $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-06T21:50:32.803Z
Answers.Text: This makes a lot of sense, thanks @matorbi
Answers.UserDisplayName: hongshan.li
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T11:38:39.877Z
Answers.LastActivityDate: 2022-11-05T11:38:39.877Z
Answers.UserDisplayName: matorbi
Body: <p>In Theorem 3.1 of the <a href="https://proceedings.neurips.cc/paper/2020/file/0d2b2061826a5df3221116a5085a6052-Paper.pdf" rel="nofollow noreferrer">conservative q-learning paper</a>, what is the meaning of</p>
  <p><span class="math-container">$$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$</span>?</p>
  <p>I thought <span class="math-container">$(I - \gamma P^{\pi})^{-1}$</span> is to be interpreted as an operator on function space over state and action space, but it is unclear to me how it is being applied to a <span class="math-container">$\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)}$</span>, which is a scalar.</p>
  <p>Also, it is unclear what <span class="math-container">$(I - \gamma P^{\pi})^{-1} \frac{C_{r, T, \delta}R_{\max}}{1 - \gamma}$</span> means?</p>
  
Tags: <reinforcement-learning><q-learning><papers>
Title: What is the meaning of $ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?
UserDisplayName: hongshan.li
ThreadText: # What is the meaning of $ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?
  
  (Asked by: username_0 on 2022-10-30)
  
  In Theorem 3.1 of the [conservative q-learning paper](https://proceedings.neurips.cc/paper/2020/file/0d2b2061826a5df3221116a5085a6052-Paper.pdf), what is the meaning of
  
  $$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi\_{\beta}(a|s)} \right](s, a)$$?
  
  I thought $(I - \gamma P^{\pi})^{-1}$ is to be interpreted as an operator on function space over state and action space, but it is unclear to me how it is being applied to a $\frac{\mu(a|s)}{\hat \pi\_{\beta}(a|s)}$, which is a scalar.
  
  Also, it is unclear what $(I - \gamma P^{\pi})^{-1} \frac{C\_{r, T, \delta}R\_{\max}}{1 - \gamma}$ means?
  
  **Tags:** reinforcement-learning, q-learning, papers
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-05. Score: 1 | Accepted answer
  
  We have for $\gamma < 1$:
  
  $$ (I-\gamma P^{\pi})^{-1}=I+\gamma P^{\pi} + \gamma^2 (P^{\pi})^2 + \cdots $$
  
  What they mean by $(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}\_{\beta}(a\mid s)}(s,a) $ is the function from $\mathcal{S}\times\mathcal{A}\longrightarrow \mathbb{R}$, sending $(s,a)\mapsto (I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}\_{\beta}(a\mid s)}$.
  
  Now, we have
  
  $$P^{\pi}\frac{\mu(a\mid s)}{\hat{\pi}\_{\beta}(a\mid s)}=\mathbb{E}\_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s')}\left[ \frac{\mu(a'\mid s')}{\hat{\pi}\_{\beta}(a'\mid s')} \right]. $$
  Similarily,
  
  $$(P^{\pi})^2\frac{\mu(a\mid s)}{\hat{\pi}\_{\beta}(a\mid s)}=\mathbb{E}\_{s'\sim p(\cdot\mid s,a), a'\sim \pi(\cdot\mid s'), s''\sim p(\cdot\mid s',a'), a''\sim \pi(\cdot\mid s'')}\left[ \frac{\mu(a''\mid s'')}{\hat{\pi}\_{\beta}(a''\mid s'')} \right]. $$
  
  Adding this together, we have
  
  $$(I-\gamma P^{\pi})^{-1}\frac{\mu(a\mid s)}{\hat{\pi}\_{\beta}(a\mid s)}(s,a)=\mathbb{E}\_{\pi}\left[\sum\_{t\geq 0} \gamma^t\frac{\mu(a\_t\mid s\_t)}{\hat{\pi}\_{\beta}(a\_t\mid s\_t)}\bigg\rvert s\_0=s, a\_0=a \right]. $$
  
  The last quantity is easier, as it does not depend on $(s,a)$, so:
  
  $$(I-\gamma P^{\pi})^{-1}\frac{C\_{r,T,\delta}R\_{\max}}{1-\gamma}=\sum\_{t\geq 0}\gamma^t\frac{C\_{r,T,\delta}R\_{\max}}{1-\gamma}=C\_{r,T,\delta}R\_{\max} $$
  
  > **Comments:**
  > **username_0**: This makes a lot of sense, thanks @matorbi  
  
  ---
  
  