Answers.Body: <p>Epsilon-greedy is one method of making an agent explore the state space to ensure that the agent doesn't settle on a sub-optimal policy. By taking random actions, even with a small probability, the agent can get to places in the state space it normally wouldn't see and on the chance that the outcome is better than what it normally would have seen, it can modify its value function (or Q function) for that state (or action) to ensure that that action is taken in the future. An alternative to Epsilon-greedy methods is modifying the policy directly using Policy Gradient methods.</p>
  <p>Eligibility traces attempt to solve a different problem. Their function is to keep a short term memory of what states have been recently visited. They unify and generalize TD and Monte Carlo methods producing a family of methods spanning a spectrum that has Monte Carlo methods at one end (<span class="math-container">$\lambda = 1$</span>) and one-step TD methods at the other (<span class="math-container">$\lambda = 0$</span>). Using traces, one can keep the features of previous states around, but at faded values depending on the choice of <span class="math-container">$\lambda$</span>. Choosing a low <span class="math-container">$\gamma$</span> (near <span class="math-container">$0$</span>) makes the traces myopic in that the traces quickly go to <span class="math-container">$0$</span>. If a larger <span class="math-container">$\lambda$</span> is used (near <span class="math-container">$1$</span>) the traces stay around for much longer and can help an agent determine the difference between what would otherwise be 2 identical states.</p>
  <p>For more information, check out <a href="https://web.archive.org/web/20171115040303/http://ufal.mff.cuni.cz/%7Estraka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a>, chapter 2 (page 30 introduces <span class="math-container">$\epsilon$</span>-greedy methods) and chapter 12 (eligibility traces).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-10-21T19:10:35.343Z
Answers.LastActivityDate: 2024-11-29T07:55:07.070Z
Answers.LastEditDate: 2024-11-29T07:55:07.070Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>I understand that in <a href="http://old.sztaki.hu/%7Eszcsaba/papers/RLAlgsInMDPs-lecture.pdf" rel="nofollow noreferrer">Reinforcement Learning algorithms</a>, such as <a href="https://en.wikipedia.org/wiki/Q-learning" rel="nofollow noreferrer">Q-learning</a>, in order to prevent selecting the actions with greatest q-values too fast and allow for exploration, we use eligibility traces.</p>
  <ol>
  <li><p>Does <span class="math-container">$\epsilon$</span>-greedy solve the same problem?</p>
  </li>
  <li><p>Do these two approaches aim to attain the same objective?</p>
  </li>
  <li><p>What are the advantages of each over the other?</p>
  </li>
  </ol>
  
Tags: <reinforcement-learning><comparison><exploration-exploitation-tradeoff><epsilon-greedy-policy><eligibility-traces>
Title: Do eligibility traces and epsilon-greedy do the same task in different ways?
UserDisplayName: Abhishek Dhyani
ThreadText: # Do eligibility traces and epsilon-greedy do the same task in different ways?
  
  (Asked by: username_0 on 2017-10-21)
  
  I understand that in [Reinforcement Learning algorithms](http://old.sztaki.hu/%7Eszcsaba/papers/RLAlgsInMDPs-lecture.pdf), such as [Q-learning](https://en.wikipedia.org/wiki/Q-learning), in order to prevent selecting the actions with greatest q-values too fast and allow for exploration, we use eligibility traces.
  
  1. Does $\epsilon$-greedy solve the same problem?
  2. Do these two approaches aim to attain the same objective?
  3. What are the advantages of each over the other?
  
  **Tags:** reinforcement-learning, comparison, exploration-exploitation-tradeoff, epsilon-greedy-policy, eligibility-traces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-10-21. Score: 3 | Accepted answer
  
  Epsilon-greedy is one method of making an agent explore the state space to ensure that the agent doesn't settle on a sub-optimal policy. By taking random actions, even with a small probability, the agent can get to places in the state space it normally wouldn't see and on the chance that the outcome is better than what it normally would have seen, it can modify its value function (or Q function) for that state (or action) to ensure that that action is taken in the future. An alternative to Epsilon-greedy methods is modifying the policy directly using Policy Gradient methods.
  
  Eligibility traces attempt to solve a different problem. Their function is to keep a short term memory of what states have been recently visited. They unify and generalize TD and Monte Carlo methods producing a family of methods spanning a spectrum that has Monte Carlo methods at one end ($\lambda = 1$) and one-step TD methods at the other ($\lambda = 0$). Using traces, one can keep the features of previous states around, but at faded values depending on the choice of $\lambda$. Choosing a low $\gamma$ (near $0$) makes the traces myopic in that the traces quickly go to $0$. If a larger $\lambda$ is used (near $1$) the traces stay around for much longer and can help an agent determine the difference between what would otherwise be 2 identical states.
  
  For more information, check out [Reinforcement Learning: An Introduction](https://web.archive.org/web/20171115040303/http://ufal.mff.cuni.cz/%7Estraka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf), chapter 2 (page 30 introduces $\epsilon$-greedy methods) and chapter 12 (eligibility traces).
  
  
  ---
  
  