Body: <p>I read this article: "<a href="https://docdro.id/2FVna0t" rel="nofollow noreferrer">Towards Autonomous Data Ferry Route Design through Reinforcement Learning</a>" by Daniel Henkel and Timothy X Brown. It specifies an infinite horizon problem where they use as a reward function for TD[0] the following:</p>
  
  <p><span class="math-container">\begin{equation}
  r(s,a) = - \int_{t_0}^{t_1} (Ft +N_0)e^{-\beta t} dt
  \end{equation}</span></p>
  
  <p>where <span class="math-container">$N_0$</span> and <span class="math-container">$F$</span> are constant, <span class="math-container">$\beta$</span> is used to adjust the discount factor and <span class="math-container">$t_0 , t_1$</span> are the initial and final time.</p>
  
  <p>Then they proceed to use <span class="math-container">$e^{-\beta t}$</span> as the <span class="math-container">$\gamma$</span> (discount factor) in the TD[0] update formula and in the policy formula.</p>
  
  <p><em>Why is the discount factor in the infinite horizon problem <span class="math-container">$e^{-\beta t}$</span>, and why is it used as <span class="math-container">$\gamma$</span> in <span class="math-container">$V(s)$</span> update, since it is a variable factor</em>?</p>
  
  <p>Also, in the formula of the TD[0] update they don't subtract <span class="math-container">$\alpha V(s)$</span>. They do:
  <span class="math-container">\begin{equation}
  V_{t+1}(s) = V_t(s) + \alpha( r(s,a) + e^{-\beta t_a} V_t(s'))
  \end{equation}</span>
  I really think this is a mistake, and the values of <span class="math-container">$V$</span> will explode without it, even in an infinite horizon problem. <em>Am I correct ? Is <span class="math-container">$- V(s)$</span> missing inside the brackets?</em></p>
  
  <p>Finally, if someone is willing and has the time to directly read this part of the article and enlighten me on if <span class="math-container">$t_0$</span> and <span class="math-container">$t_1$</span> represent the initial and final time of an action OR <span class="math-container">$t_0$</span> is always 0 and <span class="math-container">$t_1$</span> is the duration of the action, I would appreciate it. I ask this because from what is written in the paper <span class="math-container">$t_0$</span> seems to be the current time in the simulation, but I'm afraid that would just decay too fast and after some actions the reward would be close to 0. It is really not well explained and I'm a bit confused. </p>
  
  <p>Thank you for your time if you got this far reading. Any guideline answer will be very much appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-01T08:21:30.967Z
Comments.Text: You should link the article, or at least give the authors and publication, if you hope for someone to read it or refer to it in their answer. Otherwise in order to help you, someone will need to go search for the article. Use [edit] to add details.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-01T11:55:35.257Z
Comments.Text: Hi Miguel. Please, see this https://ai.meta.stackexchange.com/q/1527/2444.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><papers><temporal-difference-methods>
Title: Infinite horizon in Reinforcement Learning
UserDisplayName: Miguel Saraiva
ThreadText: # Infinite horizon in Reinforcement Learning
  
  (Asked by: username_0 on 2019-06-01)
  
  I read this article: "[Towards Autonomous Data Ferry Route Design through Reinforcement Learning](https://docdro.id/2FVna0t)" by Daniel Henkel and Timothy X Brown. It specifies an infinite horizon problem where they use as a reward function for TD[0] the following:
  
  \begin{equation}
  r(s,a) = - \int\_{t\_0}^{t\_1} (Ft +N\_0)e^{-\beta t} dt
  \end{equation}
  
  where $N\_0$ and $F$ are constant, $\beta$ is used to adjust the discount factor and $t\_0 , t\_1$ are the initial and final time.
  
  Then they proceed to use $e^{-\beta t}$ as the $\gamma$ (discount factor) in the TD[0] update formula and in the policy formula.
  
  *Why is the discount factor in the infinite horizon problem $e^{-\beta t}$, and why is it used as $\gamma$ in $V(s)$ update, since it is a variable factor*?
  
  Also, in the formula of the TD[0] update they don't subtract $\alpha V(s)$. They do:
  \begin{equation}
  V\_{t+1}(s) = V\_t(s) + \alpha( r(s,a) + e^{-\beta t\_a} V\_t(s'))
  \end{equation}
  I really think this is a mistake, and the values of $V$ will explode without it, even in an infinite horizon problem. *Am I correct ? Is $- V(s)$ missing inside the brackets?*
  
  Finally, if someone is willing and has the time to directly read this part of the article and enlighten me on if $t\_0$ and $t\_1$ represent the initial and final time of an action OR $t\_0$ is always 0 and $t\_1$ is the duration of the action, I would appreciate it. I ask this because from what is written in the paper $t\_0$ seems to be the current time in the simulation, but I'm afraid that would just decay too fast and after some actions the reward would be close to 0. It is really not well explained and I'm a bit confused.
  
  Thank you for your time if you got this far reading. Any guideline answer will be very much appreciated.
  
  **Tags:** reinforcement-learning, papers, temporal-difference-methods
  > **Comments:**
  > **username_1**: You should link the article, or at least give the authors and publication, if you hope for someone to read it or refer to it in their answer. Otherwise in order to help you, someone will need to go search for the article. Use [edit] to add details.  
  > **username_2**: Hi Miguel. Please, see this https://ai.meta.stackexchange.com/q/1527/2444.  
  
  ---
  
  ## 0 answers
  
  