Answers.Body: <p>If you really just want an SMDP-version of the algorithm, which only needs to be capable of operating on the "high-level" time scale of macro-actions, you can relatively safely take the original pseudocode of whatever MDP-based algorithm you like, replace every occurrence of "action" with "macro-action", and you're pretty much done.</p>
  
  <p>The only caveat I can think of in the case of <span class="math-container">$Q(\lambda)$</span> is that the "optimal" value for <span class="math-container">$\lambda$</span> is probably somewhat related to the amount of time that expires... so intuitively I'd expect it to be best if the value for <span class="math-container">$\lambda$</span> decreases as the amount of time expired during execution of the last macro-action increases. A constant <span class="math-container">$\lambda$</span> probably still works fine as well though.</p>
  
  <hr>
  
  <p>If you actually want your algorithm to also be aware of lower-time-scale MDP underlying an SMDP, and not only treat macro-actions as "large actions" and be done with it... I'd recommend looking into the Options framework. There you get interesting ideas like intra-option updates, which may allow you to also perform learning whilst larger macro-actions (or options) are still in progress. </p>
  
  <p>Last time I looked there hasn't been a lot of work involving the combination of eligibility traces and options, but there has been some: <a href="http://ifaamas.org/Proceedings/aamas2018/pdfs/p1008.pdf" rel="nofollow noreferrer">Eligibility Traces for Options</a>. This paper doesn't specifically apply the algorithm you mentioned (<span class="math-container">$Q(\lambda)$</span>), but it does discuss a bunch of other -- much more recent, and likely better -- off-policy algorithms with eligibility traces.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-12T19:13:29.560Z
Answers.Text: Dear @Dennis. Can I know if the only differences will be the following: 1) Take action $a$, observe $r, s^{\prime}, \tau$. 2) Updating the error: $\delta \leftarrow r+\gamma^{\tau}Q(s^{\prime}, a^{\ast})-Q(s,a)$. 3) Updating eligibility trace $e(s,a) \leftarrow \lambda \gamma^{\tau} e(s,a)$
Answers.UserDisplayName: Amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-12T19:38:06.727Z
Answers.Text: @AminSh Yes that looks correct to me, assuming that $\tau$ is the duration of the last "action" $a$ you've taken (where $a$ may be a "macro-action" that takes more than a single primitive time step). I assume that the $r$ you observe there is also a sum of rewards collected during execution of $a$, with appropriate discounting by $\gamma$ "inside" that sum as well, rather than $r$ being just a single primitive reward.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-12T19:39:08.887Z
Answers.Text: Dear @Dennis I really appreciate you.
Answers.UserDisplayName: Amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-11T17:43:24.233Z
Answers.LastActivityDate: 2019-03-11T17:43:24.233Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition. </p>
  
  <p>Can I know what are the differences between <span class="math-container">$Q(\lambda)$</span> and Q-learning for this problem (SMDP)? I actually want to extend the pseudo-code presented <a href="https://stackoverflow.com/q/40862578/3924118">here</a> to an SMDP problem with discretization of time horizon.</p>
  
Tags: <reinforcement-learning><q-learning><semi-mdp><eligibility-traces>
Title: How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs?
UserDisplayName: Amin
ThreadText: # How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs?
  
  (Asked by: username_0 on 2019-03-10)
  
  I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition.
  
  Can I know what are the differences between $Q(\lambda)$ and Q-learning for this problem (SMDP)? I actually want to extend the pseudo-code presented [here](https://stackoverflow.com/q/40862578/3924118) to an SMDP problem with discretization of time horizon.
  
  **Tags:** reinforcement-learning, q-learning, semi-mdp, eligibility-traces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-11. Score: 1
  
  If you really just want an SMDP-version of the algorithm, which only needs to be capable of operating on the "high-level" time scale of macro-actions, you can relatively safely take the original pseudocode of whatever MDP-based algorithm you like, replace every occurrence of "action" with "macro-action", and you're pretty much done.
  
  The only caveat I can think of in the case of $Q(\lambda)$ is that the "optimal" value for $\lambda$ is probably somewhat related to the amount of time that expires... so intuitively I'd expect it to be best if the value for $\lambda$ decreases as the amount of time expired during execution of the last macro-action increases. A constant $\lambda$ probably still works fine as well though.
  
  ---
  
  If you actually want your algorithm to also be aware of lower-time-scale MDP underlying an SMDP, and not only treat macro-actions as "large actions" and be done with it... I'd recommend looking into the Options framework. There you get interesting ideas like intra-option updates, which may allow you to also perform learning whilst larger macro-actions (or options) are still in progress.
  
  Last time I looked there hasn't been a lot of work involving the combination of eligibility traces and options, but there has been some: [Eligibility Traces for Options](http://ifaamas.org/Proceedings/aamas2018/pdfs/p1008.pdf). This paper doesn't specifically apply the algorithm you mentioned ($Q(\lambda)$), but it does discuss a bunch of other -- much more recent, and likely better -- off-policy algorithms with eligibility traces.
  
  > **Comments:**
  > **username_0**: Dear @Dennis. Can I know if the only differences will be the following: 1) Take action $a$, observe $r, s^{\prime}, \tau$. 2) Updating the error: $\delta \leftarrow r+\gamma^{\tau}Q(s^{\prime}, a^{\ast})-Q(s,a)$. 3) Updating eligibility trace $e(s,a) \leftarrow \lambda \gamma^{\tau} e(s,a)$  
  > **username_1**: @AminSh Yes that looks correct to me, assuming that $\tau$ is the duration of the last "action" $a$ you've taken (where $a$ may be a "macro-action" that takes more than a single primitive time step). I assume that the $r$ you observe there is also a sum of rewards collected during execution of $a$, with appropriate discounting by $\gamma$ "inside" that sum as well, rather than $r$ being just a single primitive reward.  
  > **username_0**: Dear @Dennis I really appreciate you.  
  
  ---
  
  