In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters. (Video  1 )
This paper proposes an off-policy reinforcement learning method in which the model parameters have been updated using the regression style loss function. Specifically, this method uses two regression update steps: one update value function and another one update policy using weighted regression. To compare the proposed method with others [main comprasion], 6 MuJoCo tasks are used for continuous control and LunarLander-v2 for discrete space. 
-- Even though this paper has done a good job in terms of running different experiments, the selection of some of the benchmarks seems arbitrary. For example, for discrete action space, this paper uses LunarLander which is rarely used in any papers so it makes very difficult to draw a conclusion based on these results. Common 49 Atari-2600 games should have been used for comparison. The same thing about experiments in section 5.3 is true too as those tasks are not that well-known. 
-- The proposed method doesn't outperform previous off-policy methods on Mujoco task (Table 1). Since the main claim of this paper is a new off-policy method, outperforming the previous off-policy methods is a fair game. The current results are not convincing enough.
-- There are significant overlaps between this paper, "Fitted Q-iteration by Advantage Weighted Regression", "Model-Free Preference-Based Reinforcement Learning ", and "Reinforcement learning by reward-weighted regression for operational space control" which makes the contribution of this paper very incremental.
-- The authors used only 5 seeds to run Mujoco experiments. Given the sensitivity of Mujoco for different starting points, the experiments should have been run at least with 10 different seeds.
Questions:
1) Shouldn't be an importance sampling ratio between \pi and \mu in the equations? starting from eq.5. 
2) Does the algorithm optimize the respect to $w$ as well? (eq. 15) if yes, why it is not mentioned in algorithm 1? Plus, since $d(s)$ is uniform dist. (at least this is assumed for implementation), eq. 14,15,and 16 (wherever there is d(s)), those can be simplified, e.g. \hat{V} = \sum(w_i V_i), wouldn't be better just introduced simplified version rather than current ones? (referring to the only equation above section 3.3)
3) Is this the same code used to report results in this paper? if yes, I didn't see any seed assignment in the code?! and what is "action_std" in the code?
There are a couple of recent works in merging on-policy with off-policy updates which you might want to cite them.