Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions. To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles. These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sampleefficiency. We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.
This paper proposes to reparameterize the policy using a form of ranking, and updating the policy gradient update accordingly. This on it's own isn't particularly useful, however the authors use this insight to attempt to convert the RL problem into a supervised learning problem. I think this is an interesting paper and approach, however some issues remain.
High level:
The reward threshold appears to require a-priori knowledge of the correct scale to set it at, and the maximum possible reward, which is unfortunate. Also I'm not sure it would make sense in a sparse reward environment, or an environment with small reward leading up to a single large reward, since the trajectories that get small reward might be discarded erroneously. Maybe there should be an annealing schedule for this parameter? This deserves some discussion at least.
The experimental results are very good, however you are using off-policy replay data and therefore have the issue that using the same data many times can (often) improve performance for any off-policy algorithm. This needs to be controlled and investigated very closely, however I don't even see the batch size being used listed anywhere. The authors need to really demonstrate that the benefit is coming from their algorithm and *not* just the re-use of replay data that the baselines don't get to see as much of.
More minor:
Firstly, there is very bad writing in places, .e.g, abstract: "The state-of-the-art uses action value function to derive policy". Things like this appear in many places.
Q-learning, Q-values sometimes Q is upper case sometimes lower case.
Relative action values are never defined, but if they are what I think they are then they are usually referred to as 'advantage values', and are generally given the notation of A(s,a).
Does the policy in (2) sum to 1?
Assumption 1 is written very confusingly. I would state it as something like:
"Assumption 1. For a state s and action i the set of events {E^i_j}_{j \neq i}, where E^i_j corresponds to action ai being ranked higher than action aj, are conditionally independent, given a MDP and a stationary policy."
Assumption 1: What is is conditioned on?
Thm 1: the statement of the theorem has an equality, but the proof has a clear 'approximate equality', this is very misleading since the statement is not actually true!
I don't follow the proof of corollary 5, especially the sentence explaining (65) -> (66), which doesn't parse. Again the proof is actually approximate but the statement is given using an equality, which is false.
The appendix is very messy and several parts of it just consist of series of equations with no supporting explanation.
Refs:
The original DQN paper (Mnih et al) should be cited after you mention DQN.
I was surprised not to see references to 
ACER: https://arxiv.org/abs/1611.01224 and 
PGQ: https://arxiv.org/abs/1611.01626 when discussing the off-policy / RL + SL work.
Further, in your discussion about entropy regularized RL "However, the discrepancy between
the entropy-regularized objective and original long-term reward exists due to the entropy term." - this has been corrected by a recent work: https://arxiv.org/abs/1807.09647. Probably worth mentioning it here too.