Infinite length of trajectories is an almost universal assumption in the theoretical foundations of reinforcement learning. In practice learning occurs on finite trajectories. In this paper we examine a specific result of this disparity, namely a strong bias of the time-bounded Every-visit Monte Carlo value estimator. This manifests as a vastly different learning dynamic for algorithms that use value predictors, including encouraging or discouraging exploration. We investigate these claims theoretically for a one dimensional random walk, and empirically on a number of simple environments. We use GAE as an algorithm involving a value predictor and evolution strategies as a reference point.
The paper addresses the problem of truncating trajectories in Reinforcement Learning. The scope is right for ICLR. 
The presentation is pretty good as far as the English goes but suffers from serious problems on the level of formulating concepts. Overall, I believe that the issue addressed by the paper is important, but I am not sure whether the approach taken by the authors addresses it.
I have the following complaints.
1. The paper suffers from a fundamental confusion about what problem it is trying to address. There are two straightforward ways to formulate the problem. First, we can formulate the task as solving an MDP with infinite trajectories and ask the question of what we can learn training from finite ones. The learned policies would then be evaluated on the original MDP (in practice using trajectories that are, say, an order of magnitude longer or, for a simple MDP, analytically). Second, we can consider the family of episodic MDPs parametrised by trajectory length T and ask the question of what we can learn by training on some values of T and evaluating on others. These problems are similar, but not the same and should be carefully distinguished. Right now, the introduction reads like the authors were trying to use the first approach but Section 4 reads like they are doing the second: "If during training we just used times up to T, but deployed the agent in an environment with times greater than T". Either way, the concept of bias, which appears throughout the paper, isn't formally defined anywhere. I believe that a paper that claims to address bias should have an equation that defines it as a difference between two clearly defined quantities. It should also be clearly and formally distinguished which quantities are deterministic and which are random variables. The introduction seems to (implicitly?) define bias as a random variable, section 3.1 seems to talk about "bias introduced by initializing the network". As the paper stands now, the working definition of bias used by the authors seems to be that some quantity is vaguely wrong. I do expect a higher standard of clarity in a scientific paper,
2. The paper confounds the problem of learning the value function, specifying the initial estimates of the value function and exploration. The analysis of exploration is entirely informal and suffers from the lack of clear problem formulation as per (1). Of course, one can influence exploration by initialising the value function in various ways, and this may respond differently to different truncations (different values of T), but I don't see how it is related to the "bias" problem that the paper is trying to address. In any case, I wish the authors either provided a formal handle on exploration or shift the focus of the paper and remove it,
3. I don't see what hypothesis the experiments are trying to test. Clearly, if I train my agent on a different MDP and test it on a different one, I get a mismatch. The lack of clear definitions as per (1) comes back with a vengeance.
4. Section 1.1 seems to exist purely to create a spurious impression of formality, which bears little relevance for what the paper is actually about. RL, as traditionally formulated, uses discrete time-steps so the Brownian motion model developed in this section doesn't seem very applicable - it is true that it is a limiting case of a family of discrete chains, but I don't see how this produces any insights about RL - chains are easy to simulate, so why not test on a chain directly? In any case, the result shown in Section 1.1 is entirely standard, can be found in any textbook on stochastic processes and was likely introduced purely to cover for the lack on any substantial theory that comes form the authors.
To summarize: if the authors address these points, there is a possibility that the ideas presented in this draft may somehow lead to a paper at some later point. However, I feel that the required changes would be pretty massive and don't see how the authors could make it during the ICLR revision phase - the problems aren't details or technicalities but touch the very substance of what the paper is trying to do. Basically, the whole paper would need to be re-written.
another minor point:  sloppy capitalization in section 2.1
===================
For reasons outlined in my comment below, I updated the score to 5 (for the new heavily updated version).