This work examines batch reinforcement learning-the task of maximally exploiting a given batch of off-policy data, without further data collection. We demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are only capable of learning with data correlated to their current policy, making them ineffective for most off-policy applications. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space to force the agent towards behaving on-policy with respect to a subset of the given data. We extend this notion to deep reinforcement learning, and to the best of our knowledge, present the first continuous control deep reinforcement learning algorithm which can learn effectively from uncorrelated off-policy data.
Summary:
Proposes BCRL for learning from a fixed collection of off-policy experience (I'll call this the "training data"). BCRL attempts to avoid backing up values from states that are not present in the training data, on the assumption that the current estimates of these values are likely to be inaccurate. In the continuous state-action case, this is accomplished by training a generative model to propose, given a state `s`, an action `a` such that a transition similar to `(s, a)` is in the training data. A secondary policy is then trained to perturb the proposed action within a constrained region to maximize value. BCRL outperforms DDPG and DQN when learning from fixed data, but BCRL is slightly worse than behavior cloning at learning to reproduce an expert policy that does not take exploration actions.
Review:
The overall approach is sound. The problem of extrapolation is intuitively obvious, but not something I had thought about before. I think typically exploration would correct the problem since states with over-estimated values would become more likely to be reached, giving an opportunity to get a better estimate. 
The learning setting is closer to imitation learning than to what I would call RL, since the BCRL approach essentially avoids extrapolation error by ignoring the parts of the problem that are not represented in the training data. The well-known problem with behavior cloning is compounding errors once the agent strays into areas of the state space that are far from the training data. To me "off-policy RL" implies that the goal is to learn a complete policy from off-policy data. I think the "competitors" to which BCRL should be compared are imitation learning algorithms address noisy demonstrations, and not so much off-policy RL algorithms. It would also be interesting to see the generalization performance of BCRL outside of its training data.
The BCRL idea might be applicable in a conventional RL setting as well, since the initial stages of learning could be subject to a similar extrapolation error until there has been enough exploration. A comparison to something like TRPO in this setting would be interesting.
The paper is well-written with good coverage of related literature. There are a few points where the technical content is imprecise, which I note below. 
Comments / Questions:
* Could one obtain a similar effect to BCRL by simply initializing the value estimates pessimistically?
* Sec 4.1: Since B is a set of (s, a, s', r) tuples, what does it mean for a state s' to be "in B"? Similar question for state-action tuples (s, a).
* As you note in the appendix, the construction in Sec 4.1 is essentially creating a new MDP that contains only the transitions that occur in the training data. I'd suggest stating as much in the main paper for intuition.
* Sec 4.2 / 5: The perturbation constraint \Phi is set to 0.05 in the experiments. Since the actions in these control problems are vectors, what does a scalar constraint correspond to? How is the constraint enforced during learning?
* What are the distance functions D_S and D_A?
Pros:
* A good approach to applying RL methods in the "imitation-like" setting. I've seen similar things attempted before, but this method makes more sense. 
Cons:
* The learning setting is more like "fuzzy" behavior cloning from noisy data than off-policy RL. Experimental comparison against more-sophisticated imitation learning approaches is missing.