Then R $^{(}$̂ π ) = o$_{P}$ ( n − 1 / $^{2}$) . 

Lemma 6 shows that (9) and (10) are met. Conditions (8) and (12) follow from Lemma A.7 in Appendix A and from the maximal inequality stated in [28, Lemma B.3], whose assump- tions drive the conditions on {Q$_{n}$}$_{n}$$_{≥}$$_{1}$ stated in Corollary 14. A concrete example of such a sequence can be found in [8, Section 4.4] (see also Section 4.1 therein for a brief reminder about the uniform integral entropy). Additional mild assumptions guaranteeing that an es- timator ̂ V can be defined as requested by Corollary 14 can be adapted from [8]. Finally, we note for the ERM property (13) that each π ∈ Π is known once one knows the β ∈ ∪$_{n}$$_{≥}$$_{1}$ B$_{n}$ that indexes this π . Hence, it suffices to study the estimated value ̂ V ( π ) for each such π . The appropriate algorithm for carrying out this optimization will depend on the particular choice of classes B$_{n}$ and the working model Q$_{β}$ . 

## 5 Discussion 

We have presented fast rates of regret decay in optimal policy estimation, i.e. , rates of decay that are faster than the rate of decay of the standard error of an efficient estimator of the value of any given policy in the candidate class. Our method of proof for our primary result, Theorem 1, leverages the fact that the empirical process over a Donsker class converges in distribution to a Gaussian process, and that the sample paths of this limiting process are (almost surely) uniformly continuous. The downside of our analysis, namely passing to the limit and then studying the behavior of the limiting process, is that it does not appear to allow one to obtain a faster than o$_{P}$ ( n − 1 / $^{2}$) rate of convergence for the regret. 

It would be of interest to replace our limiting argument by finite sample results that would allow one to exploit the finite sample equicontinuity of the empirical process to demonstrate faster rates. Nonetheless, we note that the problem-dependent margin condition will often have a major impact on the extent to which the rate can be improved. Furthermore, as we discussed in Section 3.2, the existence of the remainder term Rem$_{n}$ necessitates a careful 

