In reinforcement learning, it is common to let an agent interact with its environment for a fixed amount of time before resetting the environment and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed amount of time, or (ii) an indefinite period where the time limit is only used during training. In this paper, we investigate theoretically how time limits could effectively be handled in each of the two cases. In the first one, we argue that the terminations due to time limits are in fact part of the environment, and propose to include a notion of the remaining time as part of the agent's input. In the second case, the time limits are not part of the environment and are only used to facilitate learning. We argue that such terminations should not be treated as environmental ones and propose a method, specific to value-based algorithms, that incorporates this insight by continuing to bootstrap at the end of each partial episode. To illustrate the significance of our proposals, we perform several experiments on a range of environments from simple few-state transition graphs to complex control tasks, including novel and standard benchmark domains. Our results show that the proposed methods improve the performance and stability of existing reinforcement learning algorithms.
Summary: This paper explores how to handle two practical issues in reinforcement learning. The first is including time remaining in the state, for domains where episodes are cut-off before a terminal state is reached in the usual way. The second idea is to allow bootstrapping at episode boundaries, but cutting off episodes to facilitate exploration. The ideas are illustrated through several well-worked micro-world experiments.
Overall the paper is well written and polished. They slowly worked through a simple set of ideas trying to convey a better understanding to the reader, with a focus on performance of RL in practice.
My main issue with the paper is that these two topics are actually not new and are well covered by the existing RL formalisms. That is not to say that an empirical exploration of the practical implications is not of value, but that the paper would be much stronger if it was better positioned in the literature that exists.
The first idea of the paper is to include time-remaining in the state. This is of course always possible in the MDP formalism. If it was not done, as in your examples, the state would not be Markov and thus it would not be an MDP at all. In addition, the technical term for this is finite horizon MDPs (in many cases the horizon is taken to be a constant, H). It is not surprising that algorithms that take this into account do better, as your examples and experiments illustrate. The paper should make this connection to the literature more clear and discuss what is missing in our existing understanding of this case, to motivate your work. See Dynamic Programming and Optimal Control and references too it.
The second idea is that episodes may terminate due to time out, but we should include the discounted value of the time-out termination state in the return. I could not tell from the text but I assume, the next transition to the start state is fully discounted to zero, otherwise the value function would link the values of S_T and the next state, which I assume you do not want. The impact of this choice is S_T is no longer a termination state, and there is a direct fully discounted transition to the start states. This is in my view is how implementations of episodic tasks with a timeout should be done and is implemented this way is classic RL frameworks (e.g., RL glue). If we treat the value of S_T as zero or consider gamma on the transition into the time-out state as zero, then in cost to goal problems the agent will learn that these states are good and will seek them out leading to suboptimal behavior. The literature might not be totally clear about this, but it is very well discussed in a recent ICML paper: White 2017 [1]
Another way to pose and think about this problem is using the off-policy learning setting---perhaps best described in the Horde paper [2]. In this setting the behavior policy can have terminations and episodes in the classic sense (perhaps due to time outs). However, the agent's continuation function (gamma : S -> [0,1]) can specify weightings on states representing complex terminations (or not), completely independent of the behavior policy or actual state transition dynamics of the underlying MDP. To clearly establish your contributions, the authors must do a better job of relating their work to [1] and [2].
[1] White. Unifying task specification in reinforcement learning. Martha White. International Conference on Machine Learning (ICML), 2017.
[2] Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., & Precup, D. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems: 2, 761--768. 
Small comments that did not impact paper scoring:
1) eq 1 we usually don't use the superscript \gamma
2) eq2, usually we talk about truncated n-step returns include the value of the last state to correct the return. You should mention this
3) Last paragraph of page 2 should not be in the intro
4) in section 2.2 why is the behavior policy random instead of epsilon greedy?
5) It would be useful to discuss the average reward setting and how it relates to your work.
6) Fig 5. What does good performance look like in this domain. I have no reference point to understand these graphs
7) page 9, second par outlines alternative approaches but they are not presented as such. Confusing