Answers.Body: <p>The notation varies a lot, because there is no such rule on how to write it formally.</p>
  <p>The definition of the value is just the &quot;expected sum of (discounted) reward from a state given a policy&quot;, therefore the second one, in my opinion, is way clearer than the first one</p>
  <p>Yet, the first one is also not entirely wrong, but both miss the fact that the reward function is a function of the state, action and also possibly the new state, so <span class="math-container">$r(s,a,s')$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-30T13:29:06.423Z
Answers.LastActivityDate: 2024-09-30T13:29:06.423Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>Both definitions are equivalent. The point of Markov decision processes is that only the previous state has an impact on the transition dynamics, not the entire history. You could replace the <span class="math-container">$t$</span> in the first expression by a <span class="math-container">$0$</span> without running into any issues. This, in turn, would be equivalent to writing <span class="math-container">$$V_\pi(s) = \mathbb E_{a_0,s_1,\dots}\left[\sum_{l=0}^\infty \gamma^l r(s_l)\mid s_0 = s\right],$$</span> where <span class="math-container">$a_t \sim \pi(a_t \mid s_t)$</span> and <span class="math-container">$s_{t+1} \sim P(s_{t+1} \mid s_t, a_t)$</span> for <span class="math-container">$t \geq 0$</span>, which starts to resemble the second definition. How to express that the policy <span class="math-container">$\pi$</span> influences the transition dynamics wildly differs in the literature. Some use <span class="math-container">$\mathbb E_\pi[\dots]$</span>, others use <span class="math-container">$\mathbb E[\dots \mid \pi]$</span>, like the second one.</p>
  <p>The only remaining difference are the arguments to <span class="math-container">$r$</span>. As Alberto pointed out, the most common form is <span class="math-container">$r(s, a, s')$</span>, but the others are not wrong. They are in fact all equivalent and people use different definitions based on what they aim to do with it. Modeling is easier with <span class="math-container">$r(s,a,s')$</span> since it's the most versatile, but for theory, one might prefer <span class="math-container">$r(s)$</span> since it is just a function defined on the vertices of a directed graph.</p>
  <ul>
  <li>To go from <span class="math-container">$r(s)$</span> to <span class="math-container">$r(s,a)$</span>, simply define <span class="math-container">$r(s,a) = r(s)$</span> for all <span class="math-container">$a \in A$</span>. The same can be done for <span class="math-container">$r(s,a,s')$</span>.</li>
  <li>To go from <span class="math-container">$r(s,a,s')$</span> to <span class="math-container">$r(s,a)$</span>, set <span class="math-container">$r(s,a) = \mathbb E_{s'}[r(s,a,s')]$</span>.</li>
  <li>To go from <span class="math-container">$r(s,a)$</span> to <span class="math-container">$r(s)$</span>, we can extend the state space of the MDP to <span class="math-container">$S \cup (S \times A)$</span>. For instance, let's say I have some MDP with <span class="math-container">$S = \{s_1,s_2,s_3\}$</span> and <span class="math-container">$A = \{a_1, a_2\}$</span>. If I execute action <span class="math-container">$a_1$</span> in <span class="math-container">$s_1$</span> in the original MDP, I might go to <span class="math-container">$s_2$</span> with <span class="math-container">$0.5$</span> probability, and to <span class="math-container">$s_3$</span> with <span class="math-container">$0.5$</span> probability. In the new MDP, I will go to <span class="math-container">$(s_1,a_1)$</span> with probability <span class="math-container">$1$</span>, obtain my reward <span class="math-container">$r((s_1,a_1))$</span> (which now comes from the reward function of the original MDP, namely <span class="math-container">$r(s_1,a_1)$</span>), and then - regardless of which action I execute next - go to state <span class="math-container">$s_2$</span> or <span class="math-container">$s_3$</span> with probabilities <span class="math-container">$0.5$</span> each. We also set <span class="math-container">$r(s) = 0$</span> for any <span class="math-container">$s \in S$</span>.</li>
  </ul>
  <p>While the reward of a single trajectory might change with the above transformations, the expected reward of any policy should stay the same. What I laid out is not very formal, but I hope it gets the idea across.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-31T11:17:56.130Z
Answers.LastActivityDate: 2024-11-04T16:59:09.110Z
Answers.LastEditDate: 2024-11-04T16:59:09.110Z
Answers.UserDisplayName: ndieck
Body: <p>The definition of the value function in TRPO paper is</p>
  <p><span class="math-container">\begin{align}
  V_\pi(s_t) &amp;= \mathbb{E}_{a_t,s_{t+1},\ldots} \left[ \sum_{l=0}^{\infty} \gamma^l r(s_{t+l}) \right], \\[10pt]
  a_t &amp;\sim \pi(a_t|s_t), \quad s_{t+1} \sim P(s_{t+1}|s_t, a_t) \text{ for } t \geq 0.
  \end{align}</span></p>
  <p>In <a href="https://rltheorybook.github.io/rltheorybook_AJKS.pdf" rel="nofollow noreferrer">this book</a> the value function is defined as</p>
  <p><span class="math-container">\begin{equation}
  V_M^\pi(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \;\middle|\; \pi, s_0 = s\right],
  \end{equation}</span></p>
  <p>Is there any difference between those definitions or are they equivalent? It seems the first one takes into account that the trajectory is begining somewhere else and passing through state <span class="math-container">$s_t$</span>y? The second defintion take into account just trajectories that start in the desire state to compute the expectation, but that means that trajectories that pass through the state don't matter for the value function?</p>
  <p>Can anyone help me understand this?</p>
  
Tags: <reinforcement-learning><comparison><math><value-functions><notation>
Title: Is there any actual difference between these 2 definitions of a state value function?
UserDisplayName: craaaft
ThreadText: # Is there any actual difference between these 2 definitions of a state value function?
  
  (Asked by: username_0 on 2024-09-29)
  
  The definition of the value function in TRPO paper is
  
  \begin{align}
  V\_\pi(s\_t) &= \mathbb{E}\_{a\_t,s\_{t+1},\ldots} \left[ \sum\_{l=0}^{\infty} \gamma^l r(s\_{t+l}) \right], \\[10pt]
  a\_t &\sim \pi(a\_t|s\_t), \quad s\_{t+1} \sim P(s\_{t+1}|s\_t, a\_t) \text{ for } t \geq 0.
  \end{align}
  
  In [this book](https://rltheorybook.github.io/rltheorybook_AJKS.pdf) the value function is defined as
  
  \begin{equation}
  V\_M^\pi(s) = \mathbb{E}\left[\sum\_{t=0}^{\infty} \gamma^t r(s\_t, a\_t) \;\middle|\; \pi, s\_0 = s\right],
  \end{equation}
  
  Is there any difference between those definitions or are they equivalent? It seems the first one takes into account that the trajectory is begining somewhere else and passing through state $s\_t$y? The second defintion take into account just trajectories that start in the desire state to compute the expectation, but that means that trajectories that pass through the state don't matter for the value function?
  
  Can anyone help me understand this?
  
  **Tags:** reinforcement-learning, comparison, math, value-functions, notation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-09-30. Score: 1
  
  The notation varies a lot, because there is no such rule on how to write it formally.
  
  The definition of the value is just the "expected sum of (discounted) reward from a state given a policy", therefore the second one, in my opinion, is way clearer than the first one
  
  Yet, the first one is also not entirely wrong, but both miss the fact that the reward function is a function of the state, action and also possibly the new state, so $r(s,a,s')$
  
  
  ---
  
  ### Answer by username_2 on 2024-10-31. Score: 1
  
  Both definitions are equivalent. The point of Markov decision processes is that only the previous state has an impact on the transition dynamics, not the entire history. You could replace the $t$ in the first expression by a $0$ without running into any issues. This, in turn, would be equivalent to writing $$V\_\pi(s) = \mathbb E\_{a\_0,s\_1,\dots}\left[\sum\_{l=0}^\infty \gamma^l r(s\_l)\mid s\_0 = s\right],$$ where $a\_t \sim \pi(a\_t \mid s\_t)$ and $s\_{t+1} \sim P(s\_{t+1} \mid s\_t, a\_t)$ for $t \geq 0$, which starts to resemble the second definition. How to express that the policy $\pi$ influences the transition dynamics wildly differs in the literature. Some use $\mathbb E\_\pi[\dots]$, others use $\mathbb E[\dots \mid \pi]$, like the second one.
  
  The only remaining difference are the arguments to $r$. As Alberto pointed out, the most common form is $r(s, a, s')$, but the others are not wrong. They are in fact all equivalent and people use different definitions based on what they aim to do with it. Modeling is easier with $r(s,a,s')$ since it's the most versatile, but for theory, one might prefer $r(s)$ since it is just a function defined on the vertices of a directed graph.
  
  * To go from $r(s)$ to $r(s,a)$, simply define $r(s,a) = r(s)$ for all $a \in A$. The same can be done for $r(s,a,s')$.
  * To go from $r(s,a,s')$ to $r(s,a)$, set $r(s,a) = \mathbb E\_{s'}[r(s,a,s')]$.
  * To go from $r(s,a)$ to $r(s)$, we can extend the state space of the MDP to $S \cup (S \times A)$. For instance, let's say I have some MDP with $S = \{s\_1,s\_2,s\_3\}$ and $A = \{a\_1, a\_2\}$. If I execute action $a\_1$ in $s\_1$ in the original MDP, I might go to $s\_2$ with $0.5$ probability, and to $s\_3$ with $0.5$ probability. In the new MDP, I will go to $(s\_1,a\_1)$ with probability $1$, obtain my reward $r((s\_1,a\_1))$ (which now comes from the reward function of the original MDP, namely $r(s\_1,a\_1)$), and then - regardless of which action I execute next - go to state $s\_2$ or $s\_3$ with probabilities $0.5$ each. We also set $r(s) = 0$ for any $s \in S$.
  
  While the reward of a single trajectory might change with the above transformations, the expected reward of any policy should stay the same. What I laid out is not very formal, but I hope it gets the idea across.
  
  
  ---
  
  