A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient. Recently, Jin et al. (2018)  proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP. In this paper, we adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards without accessing a generative model. We show that the sample complexity of exploration of our algorithm is bounded byÕ( SA 2 (1−γ) 7 ). This improves the previously best known result ofÕ( SA 4 (1−γ) 8 ) in this setting achieved by delayed Q-learning (Strehl  et al., 2006), and matches the lower bound in terms of as well as S and A up to logarithmic factors.
Summary: This paper adapts the UCB Q-learning method to the inifinite-horizon discounted MDP setting. With an analysis similar to that of Jin et al (2018), it shows that this algorithm achieves a PAC bound of (1-gamma)^-7 |S||A|/eps^2, improving previous best-known bound (Delayed Q-learning, Strehlet et al, 2006, (1-gamma)^-8 |S||A|/eps^4) for this case.
Evaluation: As I see this paper a direct extension of that of Jin et al (2018), I am afraid I have to recommend a rejection. 
Here are some more detailed comments:
Significance: 
This paper studies the RL problem for the infinite-horizon discounted MDP setting. This is an important setting in reinforcement learning. However, the bound is not optimal as the dependence of (1-gamma) is significantly larger than the lower bound. Moreover, both the algorithm and analysis are direct extensions of that of Jin et al, I do not see a huge technique improvement. 
Technique Novelty:
As stated in the paper, the major difficulty is that the inf-horizon case does not have a set of "consecutive episodes". Therefore the "learning error at time t cannot be decomposed as errors from a set of consecutive time
steps before t, but errors from a set of non-consecutive time steps without any structure." However, I do not see a major technological innovation is needed to get around this issue. As a result, the analysis and algorithm in this paper are very similar to that of Jin et al 2018, who nearly implicitly contain the results in this paper.
Furthermore, I would think there is a (likely) very simple reduction from the inf-horizon to finite-horizon: break the inifinite horizon into episodes of length R = O((1-\gamma)^-1 log(eps^-1)). Now, although the MDP does not restart, but it can be treated as restarting at a history-dependent initial state distribution at the beginning of every episode. So, an optimal finit-horizon algorithm in this setting is at most epsilon worse than the optimal inf-horizon algorithm, no matter where/when you start. With little to no modification, we can see that Jin et al work in this setting. Thus, we obtain an algorithm for the inf-horizon as well.
A good match for this conference?
As this paper is an adaptation of a previously known Q-learning algorithm to a slightly different setting in RL, I do not see how it fits the "learning representation" paradigm. Of course, Q-function can be argued as a representation of the MDP model, but this Q-function itself is not a new concept in this paper.