Answers.Body: <p>In reinforcement learning, the apostrophe character <span class="math-container">$(')$</span> appended to a signal usually represents the signal at the next time step. For example, <span class="math-container">$s'$</span> is the state in the time step immediately after <span class="math-container">$s$</span>. In your case, <span class="math-container">$s$</span> is the state at time step <span class="math-container">$t$</span>; therefore, <span class="math-container">$s'$</span> is the state at time step <span class="math-container">$t+1$</span>. Since the signals <span class="math-container">$s$</span>, <span class="math-container">$a$</span>, and <span class="math-container">$r$</span> do not have an appended apostrophe (or any other additional notation), they are assumed to be the state, action, and reward associated with the current time step <span class="math-container">$t$</span>. To confirm, simply compare the last two equations and see how <span class="math-container">$S_t$</span> is replaced with <span class="math-container">$s$</span>, <span class="math-container">$A_t$</span> with <span class="math-container">$a$</span>, <span class="math-container">$R_t$</span> with <span class="math-container">$r$</span>, and <span class="math-container">$S_{t+1}$</span> with <span class="math-container">$s'$</span>. In summary, <strong>information about the associated time of the <span class="math-container">$s$</span>, <span class="math-container">$a$</span>, <span class="math-container">$r$</span>, and <span class="math-container">$s'$</span> signals is implicitly encoded in their written representation</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-04T17:57:02.093Z
Answers.Text: Thank you. I still have a problem about MRP Bellman equation in my comment, can you answer that? Many thanks. The key is I don't know why $v_\pi(s)$ and $v_\pi\left(s^{\prime}\right)$ can be regard as "same" in MRP, they related to rewards sum begin from $t$ and $t+1$ resptectively.
Answers.UserDisplayName: Goldhand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-04T18:01:55.657Z
Answers.Text: If possible, please edit the post to include your question inside the main body of the post. Also, note that this website has a very specific rule that each post can only have one question. If you feel that the question in your comment is separate from the question in the post, simply make another post with that second question. Thank you for posting and we look forward to more of your questions in the near future!
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-04T18:13:21.437Z
Answers.Text: Thank you. I've edited my problem. Actually they're related problems. Hope you can answer it if you like.
Answers.UserDisplayName: Goldhand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-05T22:48:59.257Z
Answers.Text: @DeepQZero Great! I like your answer. I voted for you!
Answers.UserDisplayName: Zollikofen4
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-04T17:52:05.770Z
Answers.LastActivityDate: 2023-12-04T17:52:05.770Z
Answers.UserDisplayName: DeepQZero
Body: <p><a href="https://i.sstatic.net/au9YZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/au9YZ.png" alt="enter image description here" /></a></p>
  <p>My problem: why the last formula do not contain information about time <span class="math-container">$t$</span>?
  So if <span class="math-container">$s^{\prime}=s$</span>, do we have <span class="math-container">$v_{\pi}(s) = v_{\pi}(s^{\prime})$</span>? But this is not right I guess?
  If I am right, that they are not the same, considering Bellman equation for MRP <span class="math-container">$\boldsymbol{v}=\boldsymbol{r}+\gamma \boldsymbol{P} \boldsymbol{v}$</span>. But why the left vector <span class="math-container">$\boldsymbol{v}$</span> and the right vector <span class="math-container">$\boldsymbol{v}$</span> are the same? The left one contains reward begin from <span class="math-container">$t$</span> but the right one contains reward begin from <span class="math-container">$t+1$</span>, right?</p>
  
Tags: <reinforcement-learning><machine-learning><bellman-equations>
Title: Bellman equation and inverse matrix method
UserDisplayName: Goldhand
ThreadText: # Bellman equation and inverse matrix method
  
  (Asked by: username_0 on 2023-12-04)
  
  [![enter image description here](https://i.sstatic.net/au9YZ.png)](https://i.sstatic.net/au9YZ.png)
  
  My problem: why the last formula do not contain information about time $t$?
  So if $s^{\prime}=s$, do we have $v\_{\pi}(s) = v\_{\pi}(s^{\prime})$? But this is not right I guess?
  If I am right, that they are not the same, considering Bellman equation for MRP $\boldsymbol{v}=\boldsymbol{r}+\gamma \boldsymbol{P} \boldsymbol{v}$. But why the left vector $\boldsymbol{v}$ and the right vector $\boldsymbol{v}$ are the same? The left one contains reward begin from $t$ but the right one contains reward begin from $t+1$, right?
  
  **Tags:** reinforcement-learning, machine-learning, bellman-equations
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-04. Score: 4
  
  In reinforcement learning, the apostrophe character $(')$ appended to a signal usually represents the signal at the next time step. For example, $s'$ is the state in the time step immediately after $s$. In your case, $s$ is the state at time step $t$; therefore, $s'$ is the state at time step $t+1$. Since the signals $s$, $a$, and $r$ do not have an appended apostrophe (or any other additional notation), they are assumed to be the state, action, and reward associated with the current time step $t$. To confirm, simply compare the last two equations and see how $S\_t$ is replaced with $s$, $A\_t$ with $a$, $R\_t$ with $r$, and $S\_{t+1}$ with $s'$. In summary, **information about the associated time of the $s$, $a$, $r$, and $s'$ signals is implicitly encoded in their written representation**.
  
  > **Comments:**
  > **username_0**: Thank you. I still have a problem about MRP Bellman equation in my comment, can you answer that? Many thanks. The key is I don't know why $v_\pi(s)$ and $v_\pi\left(s^{\prime}\right)$ can be regard as "same" in MRP, they related to rewards sum begin from $t$ and $t+1$ resptectively.  
  > **username_1**: If possible, please edit the post to include your question inside the main body of the post. Also, note that this website has a very specific rule that each post can only have one question. If you feel that the question in your comment is separate from the question in the post, simply make another post with that second question. Thank you for posting and we look forward to more of your questions in the near future!  
  > **username_0**: Thank you. I've edited my problem. Actually they're related problems. Hope you can answer it if you like.  
  > **username_2**: @DeepQZero Great! I like your answer. I voted for you!  
  
  ---
  
  