Two main families of reinforcement learning algorithms, Q-learning and policy gradients, have recently been proven to be equivalent when using a softmax relaxation on one part, and an entropic regularization on the other. We relate this result to the well-known convex duality of Shannon entropy and the softmax function. Such a result is also known as the Donsker-Varadhan formula. This provides a short proof of the equivalence. We then interpret this duality further, and use ideas of convex analysis to prove a new policy inequality relative to soft Q-learning.
• Policy gradients (V. ), looks to maximize the expected reward by improving policies to favor high-reward actions. In general, the target loss function is regularized by the addition of an entropic functional for the policy. This makes policies more diffuse and less likely to yield degenerate results.
A critical step in the theoretical understanding of the field has been a smooth relaxation of the greedy max operation involved in selecting actions, turned into a Boltzmann softmax O. Nachum & Schuurmans. (2017b.). This new context has lead to a breakthrough this year J. Schulman & Abbeel. (2017) with the proof of the equivalence of both methods of Q-learning and policy gradients. While that result is extremely impressive in its unification, we argue that it is critical to look additionally at the fundamental reasons as to why it occurs. We believe that the convexity of the entropy functional used for policy regularization is at the root of the phenomenon, and that (Lagrangian) duality can be exploited as well, either yielding faster proofs, or further understanding. The contributions of our paper are as follows:
1. We show how convex duality expedites the proof of the equivalence between soft Qlearning and softmax entropic policy gradients -heuristically in the general case, rigorously in the bandit case.
2. We introduce a transportation inequality that relates the expected optimality gap of any policy with its Kullback-Leibler divergence to the optimal policy.
We describe our notations here. Abusing notation heavily by identifying measures with their densities as in dπ(a|s) = π(a|s)da, if we note as either r(s, a) or r(a, s) the reward obtained by taking action a in state s, the expected reward expands as:
K r is a linear functional of π. Adding Shannon entropic regularization 1 improves numerical stability of the algorithm, and prevents early convergence to degenerate solutions. Noting regularization strength β, the objective becomes a free energy functional, named by analogy with a similar quantity in statistical mechanics:
Crucially, viewed as a functional of π, J is convex and is the sum of two parts
2 THE GIBBS VARIATIONAL PRINCIPLE FOR POLICY EVALUATION
Summary
*******
The paper provides a collection of existing results in statistics.
Comments
********
Page 1: references to Q-learning and Policy-gradients look awkwardly recent, given that these have been around for several decades.
I dont get what is the novelty in this paper. There is no doubt that all the tools that are detailed here are extremely useful and powerful results in mathematical statistics. But they are all known.
The Gibbs variational principle is folklore, Proposition 1,2 are available in all good text books on the topic, 
and Proposition 4 is nothing but a transportation Lemma.
Now, Proposition 3 is about soft-Bellman operators. This perhaps is less standard because contraction property of soft-Bellman operator in infinite norm is more recent than for Bellman operators.
But as mentioned by the authors, this is not new either. 
Also I don't really see the point of providing the proofs of these results in the main material, and not for instance in appendix, as there is no novelty either in the proof techniques.
I don't get the sentence "we have restricted so far the proof in the bandit setting": bandits are not even mentioned earlier.
Decision
********
I am sorry but unless I missed something (that then should be clarified) this seems to be an empty paper: Strong reject.