Body: <p>I am having some trouble with answering the following question:</p>
  <p>A rat is involved in an experiment. It experiences one episode. At the first step it hears a bell. At the second step it sees a light. At the third step it both hears a bell and sees a light. It then receives some food, worth +1 reward, and the episode terminates on the fourth step. All other rewards were zero. The experiment is undiscounted (<span class="math-container">$\gamma = 1$</span>).
  Approximate the state-value function by a linear combination of these features with two parameters: <span class="math-container">$b · bell(s) + l · light(s)$</span>. If <span class="math-container">$b = 2$</span> and <span class="math-container">$l = −2$</span>, write down the sequence of <span class="math-container">$\lambda$</span>-returns <span class="math-container">$v_t^\lambda$</span> corresponding to this episode, for <span class="math-container">$\lambda = 0.5$</span>.</p>
  <p>The provided answer is this:</p>
  <p><span class="math-container">$v^\lambda_1 = 0.5 (−2 + 0.5×0 + 0.5×1) = −3/4$</span></p>
  <p><span class="math-container">$v^\lambda_2 = 0.5 (0 + 1×1) = 1/2$</span></p>
  <p><span class="math-container">$v^\lambda_3 = 0.5 (2×1) = 1$</span></p>
  <p>However, these are my calculations:</p>
  <p><span class="math-container">$v^\lambda_1 = 0.5 (.5^0×−2 + .5^1×0 + .5^2×1) = 0.875$</span></p>
  <p><span class="math-container">$v^\lambda_2 = 0.5 (.5^0×0 + .5^1×1) = 1/4$</span></p>
  <p><span class="math-container">$v^\lambda_3 = 0.5 (.5^0×1) = 1/2$</span></p>
  <p>Any help is appreciated!</p>
  <p>Edit:
  These are the functions used:</p>
  <p><span class="math-container">$v^{(n)}_t = r_{t+1} +\gamma r_{t+2}+\dots+\gamma^{n−1} r_{t+n} + \gamma^n V_{(s_{t+n})} $</span></p>
  <p><span class="math-container">$v^\lambda _t = (1−\lambda)\sum_{n=1}^\infty \lambda^{n−1} v^{(n)}_t$</span></p>
  <p>bell state = <span class="math-container">$[1 \ \ 0]^T$</span></p>
  <p>light state = <span class="math-container">$[0 \  \ 1]^T$</span></p>
  <p>bell and light state = <span class="math-container">$[1 \ \ 1]^T$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-17T00:41:51.797Z
Comments.Text: Instead of writing "TD ... question", can you please just ask your specific question in the title? Thanks
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-18T17:23:57.773Z
Comments.Text: Could you clarify what formula you are applying in your calculations please, by adding its algebraic form? Most likely you have the wrong formula for the TD plus linear state value approximation, but you could also be making incorrect substitutions.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T10:12:43.480Z
Comments.Text: @NeilSlater I've added the formulas
Comments.UserDisplayName: Nat
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T12:24:57.500Z
Comments.Text: Thanks. I don't see the linear approximator in your formulae? Typically the trace works directly on the approximator's state description vector.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-22T06:29:15.447Z
Comments.Text: That's not what I meant. The trace doesn't operate on $v$ directly, it operates on $x$, the state description vector - sometimes noted as $\Phi(x)$. So you will end up with an eligibility trace vector that does not represent a specific state, that you will multiply by $[2, -2]$ to getthe current value approximation. You also have to decide between replacing traces and accumulating traces.
Comments.UserDisplayName: Neil Slater
Tags: <temporal-difference-methods>
Title: Calculating TD($\lambda$) returns, reinforcement learning
UserDisplayName: Nat
ThreadText: # Calculating TD($\lambda$) returns, reinforcement learning
  
  (Asked by: username_0 on 2024-04-16)
  
  I am having some trouble with answering the following question:
  
  A rat is involved in an experiment. It experiences one episode. At the first step it hears a bell. At the second step it sees a light. At the third step it both hears a bell and sees a light. It then receives some food, worth +1 reward, and the episode terminates on the fourth step. All other rewards were zero. The experiment is undiscounted ($\gamma = 1$).
  Approximate the state-value function by a linear combination of these features with two parameters: $b · bell(s) + l · light(s)$. If $b = 2$ and $l = −2$, write down the sequence of $\lambda$-returns $v\_t^\lambda$ corresponding to this episode, for $\lambda = 0.5$.
  
  The provided answer is this:
  
  $v^\lambda\_1 = 0.5 (−2 + 0.5×0 + 0.5×1) = −3/4$
  
  $v^\lambda\_2 = 0.5 (0 + 1×1) = 1/2$
  
  $v^\lambda\_3 = 0.5 (2×1) = 1$
  
  However, these are my calculations:
  
  $v^\lambda\_1 = 0.5 (.5^0×−2 + .5^1×0 + .5^2×1) = 0.875$
  
  $v^\lambda\_2 = 0.5 (.5^0×0 + .5^1×1) = 1/4$
  
  $v^\lambda\_3 = 0.5 (.5^0×1) = 1/2$
  
  Any help is appreciated!
  
  Edit:
  These are the functions used:
  
  $v^{(n)}\_t = r\_{t+1} +\gamma r\_{t+2}+\dots+\gamma^{n−1} r\_{t+n} + \gamma^n V\_{(s\_{t+n})} $
  
  $v^\lambda \_t = (1−\lambda)\sum\_{n=1}^\infty \lambda^{n−1} v^{(n)}\_t$
  
  bell state = $[1 \ \ 0]^T$
  
  light state = $[0 \ \ 1]^T$
  
  bell and light state = $[1 \ \ 1]^T$
  
  **Tags:** temporal-difference-methods
  > **Comments:**
  > **username_1**: Instead of writing "TD ... question", can you please just ask your specific question in the title? Thanks  
  > **username_2**: Could you clarify what formula you are applying in your calculations please, by adding its algebraic form? Most likely you have the wrong formula for the TD plus linear state value approximation, but you could also be making incorrect substitutions.  
  > **username_0**: @NeilSlater I've added the formulas  
  > **username_2**: Thanks. I don't see the linear approximator in your formulae? Typically the trace works directly on the approximator's state description vector.  
  > **username_2**: That's not what I meant. The trace doesn't operate on $v$ directly, it operates on $x$, the state description vector - sometimes noted as $\Phi(x)$. So you will end up with an eligibility trace vector that does not represent a specific state, that you will multiply by $[2, -2]$ to getthe current value approximation. You also have to decide between replacing traces and accumulating traces.  
  
  ---
  
  ## 0 answers
  
  