Body: <p>How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with <span class="math-container">$2\epsilon t$</span>.</p>
  <p><a href="https://www.youtube.com/watch?v=LtAt5M_a0dI&amp;list=PL_iWQOsE6TfXxKgI1GgyV1B_Xa0DxE5eH&amp;index=45" rel="nofollow noreferrer">https://www.youtube.com/watch?v=LtAt5M_a0dI&amp;list=PL_iWQOsE6TfXxKgI1GgyV1B_Xa0DxE5eH&amp;index=45</a>
  <a href="https://i.sstatic.net/k2hU2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/k2hU2.png" alt="enter image description here" /></a></p>
  <p>I agree with the
  <span class="math-container">$$\sum_{s_t}p_{\theta'}(s_t)f(s_t) \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - |p_{\theta}(s_t) - p_{\theta'}(s_t)|max_{s_t}f(s_t)\big]$$</span>
  <span class="math-container">$$\because \sum_{s_t}p_{\theta}(s_t)f(s_t) = \sum_{s_t}\big(p_{\theta}(s_t) + p_{\theta'}(s_t) - p_{\theta'}(s_t) \big)f(s_t)$$</span>
  <span class="math-container">$$= \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big(p_{\theta}(s_t) - p_{\theta'}(s_t) \big)f(s_t)$$</span>
  <span class="math-container">$$ \leq \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big|p_{\theta}(s_t) - p_{\theta'}(s_t) \big|f(s_t)$$</span>
  <span class="math-container">$$ \leq \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big|p_{\theta}(s_t) - p_{\theta'}(s_t) \big|max_{s_t}f(s_t)$$</span></p>
  <p>so,
  <span class="math-container">$$\sum_{s_t}p_{\theta'}(s_t)f(s_t) \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - |p_{\theta}(s_t) - p_{\theta'}(s_t)|max_{s_t}f(s_t)\big] \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - 2\epsilon t max_{s_t}f(s_t)\big] = E_{p_{\theta}(s_t)}[f(s_t)] - \sum_{s_t}2\epsilon t max_{s_t}f(s_t)$$</span></p>
  <p>Here is the question. How does the lecture note remove the summation of the epsilon related term? The professor sasy, since all of them are constant (I understand <span class="math-container">$max_{s_t}f(s_t)$</span> is also a constant), we can write like that, but <span class="math-container">$\sum_{t=1}^{10}(t - 1) = 45 = \sum_{t=1}^{10}t - \sum_{t=1}^{10}1 = 55 - 10 = 45$</span>. We don't write <span class="math-container">$\sum_{t=1}^{10}1 = 1$</span> because 1 is a constant.</p>
  <p>The abiguous part is there is no concrete size of <span class="math-container">$s_t$</span> in the summation, but at least I think just removing the summation in front of the epsilon term is weird.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-25T15:54:27.243Z
Comments.Text: I would agree with you here, but I think the "trick" is that the first $\epsilon$ and the second are different, so for example if there are $N$ states (there is finite number of states) then the last sum would be $2 \epsilon_1 t N \max_{s_t} f(s_t)$ but you can replace this with $\epsilon_2 = N\epsilon_1$, and since $\epsilon_1$ can be picked arbitrarily small so will be $\epsilon_2$. I think the final result isn't affected but the process to get it is a bit suspicious here.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-25T17:24:06.880Z
Comments.Text: Thank you for your reply! Totally make sense for the finite case ($|s_t
  | = N$). Do you have any thought for the infinite case as well? So, all of the summation will be replaced with integral, then it's little bit tricky to just say $2\epsilon_1 t N max_{s_t}f(s_t)$.
Comments.UserDisplayName: shashack
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-26T11:38:26.857Z
Comments.Text: I think it wouldn't be even defined in case of infinite number of states. You would have something like $\lim_{(x, y) \rightarrow (0, \infty)} xy$ which isn't defined
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-26T20:55:50.727Z
Comments.Text: Hmm, I don't understand why you said $lim_{(x,y)}$. $\int_{s_t}p_{\theta'}(s_t)f(s_t) ds_t \geq E_{p_{\theta}(s_t)}[f(s_t)] - \int_{s_t}2\epsilon t max_{s_t}f(s_t) ds_t $ For the last term, it's $\int_{s_t}2\epsilon t max_{s_t}f(s_t) ds_t = 2\epsilon t max_{s_t}f(s_t)\int_{s_t} 1 ds_t$ Do you mean $\int_{s_t} 1  ds_t$ is not even defined?
Comments.UserDisplayName: shashack
Tags: <reinforcement-learning><machine-learning><trust-region-policy-optimization><total-variational-distance>
Title: CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)
UserDisplayName: shashack
ThreadText: # CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)
  
  (Asked by: username_0 on 2022-06-25)
  
  How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$.
  
  <https://www.youtube.com/watch?v=LtAt5M_a0dI&list=PL_iWQOsE6TfXxKgI1GgyV1B_Xa0DxE5eH&index=45>
  [![enter image description here](https://i.sstatic.net/k2hU2.png)](https://i.sstatic.net/k2hU2.png)
  
  I agree with the
  $$\sum\_{s\_t}p\_{\theta'}(s\_t)f(s\_t) \geq \sum\_{s\_t}\big[p\_{\theta}(s\_t)f(s\_t) - |p\_{\theta}(s\_t) - p\_{\theta'}(s\_t)|max\_{s\_t}f(s\_t)\big]$$
  $$\because \sum\_{s\_t}p\_{\theta}(s\_t)f(s\_t) = \sum\_{s\_t}\big(p\_{\theta}(s\_t) + p\_{\theta'}(s\_t) - p\_{\theta'}(s\_t) \big)f(s\_t)$$
  $$= \sum\_{s\_t}p\_{\theta'}(s\_t)f(s\_t) + \big(p\_{\theta}(s\_t) - p\_{\theta'}(s\_t) \big)f(s\_t)$$
  $$ \leq \sum\_{s\_t}p\_{\theta'}(s\_t)f(s\_t) + \big|p\_{\theta}(s\_t) - p\_{\theta'}(s\_t) \big|f(s\_t)$$
  $$ \leq \sum\_{s\_t}p\_{\theta'}(s\_t)f(s\_t) + \big|p\_{\theta}(s\_t) - p\_{\theta'}(s\_t) \big|max\_{s\_t}f(s\_t)$$
  
  so,
  $$\sum\_{s\_t}p\_{\theta'}(s\_t)f(s\_t) \geq \sum\_{s\_t}\big[p\_{\theta}(s\_t)f(s\_t) - |p\_{\theta}(s\_t) - p\_{\theta'}(s\_t)|max\_{s\_t}f(s\_t)\big] \geq \sum\_{s\_t}\big[p\_{\theta}(s\_t)f(s\_t) - 2\epsilon t max\_{s\_t}f(s\_t)\big] = E\_{p\_{\theta}(s\_t)}[f(s\_t)] - \sum\_{s\_t}2\epsilon t max\_{s\_t}f(s\_t)$$
  
  Here is the question. How does the lecture note remove the summation of the epsilon related term? The professor sasy, since all of them are constant (I understand $max\_{s\_t}f(s\_t)$ is also a constant), we can write like that, but $\sum\_{t=1}^{10}(t - 1) = 45 = \sum\_{t=1}^{10}t - \sum\_{t=1}^{10}1 = 55 - 10 = 45$. We don't write $\sum\_{t=1}^{10}1 = 1$ because 1 is a constant.
  
  The abiguous part is there is no concrete size of $s\_t$ in the summation, but at least I think just removing the summation in front of the epsilon term is weird.
  
  **Tags:** reinforcement-learning, machine-learning, trust-region-policy-optimization, total-variational-distance
  
  > **Comments:**
  > **username_1**: I would agree with you here, but I think the "trick" is that the first $\epsilon$ and the second are different, so for example if there are $N$ states (there is finite number of states) then the last sum would be $2 \epsilon_1 t N \max_{s_t} f(s_t)$ but you can replace this with $\epsilon_2 = N\epsilon_1$, and since $\epsilon_1$ can be picked arbitrarily small so will be $\epsilon_2$. I think the final result isn't affected but the process to get it is a bit suspicious here.  
  > **username_0**: Thank you for your reply! Totally make sense for the finite case ($|s_t | = N$). Do you have any thought for the infinite case as well? So, all of the summation will be replaced with integral, then it's little bit tricky to just say $2\epsilon_1 t N max_{s_t}f(s_t)$.  
  > **username_1**: I think it wouldn't be even defined in case of infinite number of states. You would have something like $\lim_{(x, y) \rightarrow (0, \infty)} xy$ which isn't defined  
  > **username_0**: Hmm, I don't understand why you said $lim_{(x,y)}$. $\int_{s_t}p_{\theta'}(s_t)f(s_t) ds_t \geq E_{p_{\theta}(s_t)}[f(s_t)] - \int_{s_t}2\epsilon t max_{s_t}f(s_t) ds_t $ For the last term, it's $\int_{s_t}2\epsilon t max_{s_t}f(s_t) ds_t = 2\epsilon t max_{s_t}f(s_t)\int_{s_t} 1 ds_t$ Do you mean $\int_{s_t} 1  ds_t$ is not even defined?  
  
  ---
  
  ## 0 answers
  
  