Answers.Body: <p>Eligibility traces are not very common in deep-RL, but I suppose that if your network is not huge they might work. I think you are describing the backward view of eligibility traces since you are saying you want to keep a &quot;trace&quot; of each state visited before reaching the terminal state.</p>
  <p>In this case, computing and updating the traces to later combine them with the TD-Error could be done as follows. You would need to initialize a <span class="math-container">$\textbf{Z}$</span> matrix with the same dimensions as the weights <span class="math-container">$(\textbf{W})$</span> for each layer of your network. Then, at each time step you update the trace by simply:</p>
  <p><span class="math-container">$$ \textbf{Z}_{ij} \leftarrow \gamma\lambda \textbf{Z}{ij} + \nabla_{\textbf{W}_{ij}}O(i) $$</span></p>
  <p>where <span class="math-container">$O(i)$</span> is the output of the network. Assuming that <span class="math-container">$\textbf{O}$</span> is the last layer and either O1 is selected or O2 (but not both, as in a policy network) this is the rule. <span class="math-container">$\gamma$</span> is the usual discount factor and <span class="math-container">$\lambda$</span> the trace decay parameter. Once you have this then update the weights with the usual TD learning rule:</p>
  <p><span class="math-container">$$ \textbf{W}_{ij} \leftarrow \textbf{W}{ij} + \delta\textbf{Z}_{ij}$$</span></p>
  <p>The TD-Error is whatever you prefer (i.e. Actor-Critic or SARSA). To understand better this combined rule, set <span class="math-container">$\lambda$</span> to be zero and see how you recover the usual TD(0) learning rule. I imagine you are familiar with Sutton&amp;Barto 2018 fantastic book in RL (not deep RL). This topic is explained in chapter 12.</p>
  <p>Recall that at each state (or time step) all the information to compute these rules is available thanks to the forward pass from the input to the output.</p>
  <p>Hope this helped.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T14:20:32.547Z
Answers.LastActivityDate: 2022-03-30T13:52:26.910Z
Answers.LastEditDate: 2022-03-30T13:52:26.910Z
Answers.UserDisplayName: JFR
Body: <p>I have a Neural Network, each Neuron is made up of <em>inputs</em>, <em>weights</em>, and <em>output</em>. I have potentially multiple hidden layers. The activation function executed against the <em>output</em> is not known by the Neuron.</p>
  <p>I would like to use TD(λ) to back-propagate errors through the network as it explores options. My understanding is that this is forward looking TD(λ) because I won't know the error until I reach a terminal state, and so an eligibility trace needs to be kept for each input+weight combination as I back-propagate the error between the NNs new output based on the state-change from the last prediction and the output from the last prediction.</p>
  <p>To try and modularise my code as much as possible, the neuron won't know the loss function, but will instead be given the error as a derivative of whatever the activation function of its output was. It also won't know if it's in a hidden layer or not, it will just have inputs, weights, and an output</p>
  <p>For example:</p>
  <p><a href="https://i.sstatic.net/02xoD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/02xoD.png" alt="enter image description here" /></a></p>
  <p>So when each Neuron is back-propagating the error signal from all its output connections (summed before it receives it), how do I calculate the eligibility trace?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T09:56:54.340Z
Comments.Text: What do you mean by "The activation function executed against the output is not known by the Neuron."? How is this possible? Also, you say "I would like to use TD(λ) to back-propagate errors through the network as it explores options.", but what do you mean by "as it explores options"? You're not talking about options in hierarchical RL, I suppose.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T11:23:11.367Z
Comments.Text: Q1: it's an implementation detail. The code for a "neuron" is unaware of the activation function used. As far as I understand it, this shouldn't matter because the "neuron" in my case will have available the output of the network as well as the derivative of it's output.
    
  Q2: "explores" options I mean to say the NN will make predictions on what to do next, and the change in state that prediction results in is the next input the next epoch
Comments.UserDisplayName: NeomerArcana
Tags: <neural-networks><reinforcement-learning><temporal-difference-methods><td-lambda>
Title: When using TD(λ), how do you calculate the eligibility trace per input & weight of a neural network neuron?
UserDisplayName: NeomerArcana
ThreadText: # When using TD(λ), how do you calculate the eligibility trace per input & weight of a neural network neuron?
  
  (Asked by: username_0 on 2022-03-24)
  
  I have a Neural Network, each Neuron is made up of *inputs*, *weights*, and *output*. I have potentially multiple hidden layers. The activation function executed against the *output* is not known by the Neuron.
  
  I would like to use TD(λ) to back-propagate errors through the network as it explores options. My understanding is that this is forward looking TD(λ) because I won't know the error until I reach a terminal state, and so an eligibility trace needs to be kept for each input+weight combination as I back-propagate the error between the NNs new output based on the state-change from the last prediction and the output from the last prediction.
  
  To try and modularise my code as much as possible, the neuron won't know the loss function, but will instead be given the error as a derivative of whatever the activation function of its output was. It also won't know if it's in a hidden layer or not, it will just have inputs, weights, and an output
  
  For example:
  
  [![enter image description here](https://i.sstatic.net/02xoD.png)](https://i.sstatic.net/02xoD.png)
  
  So when each Neuron is back-propagating the error signal from all its output connections (summed before it receives it), how do I calculate the eligibility trace?
  
  **Tags:** neural-networks, reinforcement-learning, temporal-difference-methods, td-lambda
  > **Comments:**
  > **username_1**: What do you mean by "The activation function executed against the output is not known by the Neuron."? How is this possible? Also, you say "I would like to use TD(λ) to back-propagate errors through the network as it explores options.", but what do you mean by "as it explores options"? You're not talking about options in hierarchical RL, I suppose.  
  > **username_0**: Q1: it's an implementation detail. The code for a "neuron" is unaware of the activation function used. As far as I understand it, this shouldn't matter because the "neuron" in my case will have available the output of the network as well as the derivative of it's output.    Q2: "explores" options I mean to say the NN will make predictions on what to do next, and the change in state that prediction results in is the next input the next epoch  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-29. Score: 1
  
  Eligibility traces are not very common in deep-RL, but I suppose that if your network is not huge they might work. I think you are describing the backward view of eligibility traces since you are saying you want to keep a "trace" of each state visited before reaching the terminal state.
  
  In this case, computing and updating the traces to later combine them with the TD-Error could be done as follows. You would need to initialize a $\textbf{Z}$ matrix with the same dimensions as the weights $(\textbf{W})$ for each layer of your network. Then, at each time step you update the trace by simply:
  
  $$ \textbf{Z}\_{ij} \leftarrow \gamma\lambda \textbf{Z}{ij} + \nabla\_{\textbf{W}\_{ij}}O(i) $$
  
  where $O(i)$ is the output of the network. Assuming that $\textbf{O}$ is the last layer and either O1 is selected or O2 (but not both, as in a policy network) this is the rule. $\gamma$ is the usual discount factor and $\lambda$ the trace decay parameter. Once you have this then update the weights with the usual TD learning rule:
  
  $$ \textbf{W}\_{ij} \leftarrow \textbf{W}{ij} + \delta\textbf{Z}\_{ij}$$
  
  The TD-Error is whatever you prefer (i.e. Actor-Critic or SARSA). To understand better this combined rule, set $\lambda$ to be zero and see how you recover the usual TD(0) learning rule. I imagine you are familiar with Sutton&Barto 2018 fantastic book in RL (not deep RL). This topic is explained in chapter 12.
  
  Recall that at each state (or time step) all the information to compute these rules is available thanks to the forward pass from the input to the output.
  
  Hope this helped.
  
  
  ---
  
  