In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, much recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.
This paper proposes a unifying framework, BRAC, which summarizes the idea and evaluates the effectiveness of recently proposed offline reinforcement learning algorithms, specifically BEAR, BCQ, and KL control. The authors generalize existing offline RL approaches to an actor-critic algorithm that regularizes the learned policy such that it can stay close to the behavior policy. Based on prior work, the authors state that there are two variants of regularizations to the behavior policy, value penalty (vp) and policy regularization (pr) and three choices of divergence functions along with their sample estimate that measures the distance between the learned policy and the behavior policy. The paper empirically investigates the effectiveness of each regularization scheme as well as each divergence function and conclude that vp is slightly more effective than pr while all divergence functions have similar performances.
Overall, this paper could be an interesting summary of prior works in offline RL and provide some empirical insights on the effectiveness of each building block in the previous approaches, though it neither offers theoretical explanations nor proposes a new offline RL algorithm that outperforms the existing methods under the BRAC framework. My score would be increased given some technical insights or some promising results in a relatively novel offline RL algorithm in the author’s response. 
Specifically, the paper does a thorough ablation study on BEAR, BCQ, and KL-control within the BRAC framework. While the experimental results demonstrate some interesting phenomenons such as combining vp and the primal form of KL divergence achieving the best performance and taking minimum over Q functions outperforming using a mix of maximum and minimum over the Q functions, I believe the paper would be greatly improved if the authors can provide a new offline RL method based on the BRAC that can achieve better performance than current approaches and is less incremental than simply combining vp and KL divergence. 
Though BRAC summarizes offline RL methods in a neat way, it would be more technically sound if a general theoretical analysis/insights of offline RL algorithms can be offered in the paper, e.g. showing the reason that vp is outperforming pr through convergence analysis in the tabular case.
Minor comment:
Error bars should be added to the all the bar plots.
**UPDATE**
After reading the rebuttal, I think my concerns are addressed and thus I updated my rating.