Answers.Body: <p>A normal way to deal with training on infinite horizon (aka &quot;continuing&quot; or &quot;non-episodic&quot;) problems is to use TD learning or other bootstrapping methods (of which Q-learning in DQN is one example), and to treat the cutoff at <span class="math-container">$T$</span> for pseudo-episodes as a training artefact.</p>
  <p>If the state at time <span class="math-container">$T$</span> was really a terminal state, the TD target would be just <span class="math-container">$r$</span> because <span class="math-container">$q(s^T,\cdot) = 0$</span> by definition, but that doesn't apply in your case.</p>
  <p>So always use the bootstrapped TD target - e.g. <span class="math-container">$r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$</span> for single step TD target with a Q estimate having <span class="math-container">$\theta$</span> as learned params - and don't treat the horizon data any differently.</p>
  <p>If you do this, then your concerns about under-estimates should not be an issue. Your pseudo-episodes do need to be long enough to observe the impact of multiple requests in the long term is the main issue (setting <span class="math-container">$T$</span> too low so that the system does not reach any kind of equilibrium would be a problem).</p>
  <p><a href="https://link.springer.com/article/10.1007/BF00114727" rel="nofollow noreferrer">You could also use an average reward setting</a> and differential value functions. It is slightly better from a theoretical standpoint, but Q-learning and DQN is fine if you don't want to be bothered with that. The same basic rule applies - ignore &quot;episode end&quot; for constructing TD targets because it is just a training artefact. Also ensure you set <span class="math-container">$T$</span> high enough that the long term impacts of a policy are observable.</p>
  <p>If your starting state is special (e.g. cars all in fixed places, and no requests in progress) this could also be an issue, because the real world system will rarely be in that state but you will have many episode starts with it. If you cannot start the system in a reasonable random initial state for your problem, you may also want to discard data from episode starts and allow a run-in time before using data for training from the pseudo-episodes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T13:27:55.327Z
Answers.Text: For infinite horizon, i.e. non-episodic tasks, can I use DQN with no episodes, γ=0.99, and only one terminal state that is the timestep limit ?
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T14:54:18.067Z
Answers.Text: @ddaedalus: Yes, sort of, that is what this answer describes. Technically the termination condition is not real or part of the state, so you just use it to trigger a stop to collecting training data and re-set the state for convenience. You should use value function updates as if that last state is *not* terminal. Because it isn't.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-15T07:52:58.413Z
Answers.Text: Awesome point - **ignore "episode end" for constructing TD targets because it is just a training artefact.** Helping me understand instantly, thanks. Found your solution agrees with [this paper](https://arxiv.org/abs/1712.00378) I was reading. It is all about time limits that are not regarded terminal states from the perspective of the environment. However, it involves overwriting the agent training processes, kind of tricky since I am using agents from [Stable Baselines](https://stable-baselines.readthedocs.io/en/master/). Do you have any experience for such a modification?
Answers.UserDisplayName: Guoyang Qin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-15T07:58:13.673Z
Answers.Text: Your remark about the lengh of $T$ is also helpful. However, my system is time-dependent since it is a traffic system, supply and demand are constantly changing and stochastic. which means the system is non-stationary. For a non-stationary system, if I train the agent in $[0, T]$, does it mean the policy is only applicable for $[0, T]$ since the traffic conditions change after $T$?
Answers.UserDisplayName: Guoyang Qin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-15T08:26:57.553Z
Answers.Text: @GuoyangQin It depends on what you mean by "non-stationary" and "time dependent". If rates of traffic depend reliably on time of day for instance, make that a feature in the state. If conditions are otherwise detectable or predictable through some feature, make *that* feature part of the state. If you want to deploy a trained agent that can cope with varying traffic conditions, you should train it to be optimal over those variations in conditions.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-15T08:29:03.523Z
Answers.Text: @GuoyangQin Finally, one alternative if you really do mean "non-stationary" as opposed to "depends on more factors that could become part of the state", then one solution is to deploy the system in a continuously learning setup. If this is important to you, then you could ask a new question about it
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-10T12:53:08.453Z
Answers.LastActivityDate: 2020-09-10T13:00:27.650Z
Answers.LastEditDate: 2020-09-10T13:00:27.650Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am modelling a ride-hailing system where passenger requests continuously arrive into the system. An RL model is developed to learn how to match those requests with drivers efficiently.</p>
  <p>Basically, the system can run infinitely as long as there are requests arriving (infinite horizon reality). However, in order for the RL training to conduct, the episode length should be restricted to some finite duration, say <span class="math-container">$[0,T]$</span> (finite horizon training).</p>
  <blockquote>
  <p>My question is how to implement the learned policy based on finite horizon <span class="math-container">$[0,T]$</span> to the real system with infinite horizon <span class="math-container">$[0,\infty]$</span>?</p>
  </blockquote>
  <p>I expect there would be a conflict of objectives. The value function near <span class="math-container">$T$</span> is partially cut off in a finite horizon and would become an underestimate and affect policy performance in an infinite horizon implementation. To this end, I doubt the applicability of the learned policy.</p>
  
Tags: <machine-learning><reinforcement-learning>
Title: How to implement RL policies learned on a finite horizon?
UserDisplayName: Guoyang Qin
ThreadText: # How to implement RL policies learned on a finite horizon?
  
  (Asked by: username_0 on 2020-09-10)
  
  I am modelling a ride-hailing system where passenger requests continuously arrive into the system. An RL model is developed to learn how to match those requests with drivers efficiently.
  
  Basically, the system can run infinitely as long as there are requests arriving (infinite horizon reality). However, in order for the RL training to conduct, the episode length should be restricted to some finite duration, say $[0,T]$ (finite horizon training).
  
  > My question is how to implement the learned policy based on finite horizon $[0,T]$ to the real system with infinite horizon $[0,\infty]$?
  
  I expect there would be a conflict of objectives. The value function near $T$ is partially cut off in a finite horizon and would become an underestimate and affect policy performance in an infinite horizon implementation. To this end, I doubt the applicability of the learned policy.
  
  **Tags:** machine-learning, reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-10. Score: 2 | Accepted answer
  
  A normal way to deal with training on infinite horizon (aka "continuing" or "non-episodic") problems is to use TD learning or other bootstrapping methods (of which Q-learning in DQN is one example), and to treat the cutoff at $T$ for pseudo-episodes as a training artefact.
  
  If the state at time $T$ was really a terminal state, the TD target would be just $r$ because $q(s^T,\cdot) = 0$ by definition, but that doesn't apply in your case.
  
  So always use the bootstrapped TD target - e.g. $r + \gamma \text{max}\_{a'} \hat{q}(s',a',\theta)$ for single step TD target with a Q estimate having $\theta$ as learned params - and don't treat the horizon data any differently.
  
  If you do this, then your concerns about under-estimates should not be an issue. Your pseudo-episodes do need to be long enough to observe the impact of multiple requests in the long term is the main issue (setting $T$ too low so that the system does not reach any kind of equilibrium would be a problem).
  
  [You could also use an average reward setting](https://link.springer.com/article/10.1007/BF00114727) and differential value functions. It is slightly better from a theoretical standpoint, but Q-learning and DQN is fine if you don't want to be bothered with that. The same basic rule applies - ignore "episode end" for constructing TD targets because it is just a training artefact. Also ensure you set $T$ high enough that the long term impacts of a policy are observable.
  
  If your starting state is special (e.g. cars all in fixed places, and no requests in progress) this could also be an issue, because the real world system will rarely be in that state but you will have many episode starts with it. If you cannot start the system in a reasonable random initial state for your problem, you may also want to discard data from episode starts and allow a run-in time before using data for training from the pseudo-episodes.
  
  > **Comments:**
  > **username_2**: For infinite horizon, i.e. non-episodic tasks, can I use DQN with no episodes, γ=0.99, and only one terminal state that is the timestep limit ?  
  > **username_1**: @username_2: Yes, sort of, that is what this answer describes. Technically the termination condition is not real or part of the state, so you just use it to trigger a stop to collecting training data and re-set the state for convenience. You should use value function updates as if that last state is *not* terminal. Because it isn't.  
  > **username_0**: Awesome point - **ignore "episode end" for constructing TD targets because it is just a training artefact.** Helping me understand instantly, thanks. Found your solution agrees with [this paper](https://arxiv.org/abs/1712.00378) I was reading. It is all about time limits that are not regarded terminal states from the perspective of the environment. However, it involves overwriting the agent training processes, kind of tricky since I am using agents from [Stable Baselines](https://stable-baselines.readthedocs.io/en/master/). Do you have any experience for such a modification?  
  > **username_0**: Your remark about the lengh of $T$ is also helpful. However, my system is time-dependent since it is a traffic system, supply and demand are constantly changing and stochastic. which means the system is non-stationary. For a non-stationary system, if I train the agent in $[0, T]$, does it mean the policy is only applicable for $[0, T]$ since the traffic conditions change after $T$?  
  > **username_1**: @username_0 It depends on what you mean by "non-stationary" and "time dependent". If rates of traffic depend reliably on time of day for instance, make that a feature in the state. If conditions are otherwise detectable or predictable through some feature, make *that* feature part of the state. If you want to deploy a trained agent that can cope with varying traffic conditions, you should train it to be optimal over those variations in conditions.  
  > **username_1**: @username_0 Finally, one alternative if you really do mean "non-stationary" as opposed to "depends on more factors that could become part of the state", then one solution is to deploy the system in a continuously learning setup. If this is important to you, then you could ask a new question about it  
  
  ---
  
  