Sample Complexity Bounds for Iterative Stochastic Policy Optimization
This paper is concerned with robustness analysis of decision making under uncertainty. We consider a class of iterative stochastic policy optimization problems and analyze the resulting expected performance for each newly updated policy at each iteration. In particular, we employ concentration-of-measure inequalities to compute future expected cost and probability of constraint violation using empirical runs. A novel inequality bound is derived that accounts for the possibly unbounded change-of-measure likelihood ratio resulting from iterative policy adaptation. The bound serves as a high-confidence certificate for providing future performance or safety guarantees. The approach is illustrated with a simple robot control scenario and initial steps towards applications to challenging aerial vehicle navigation problems are presented.
Paper considers iterative stochastic policy optimization, and independent of how the policy is updated, the authors provide a bound on the expected cost resulting from the execution of a new stochastic policy. using observed samples from a previous policy. The bound is illustrated in an example of aerial vehicle navigation.Quality, Results suggest that original policy could be improved to achieve only a 10% predicted collision rate (down from 50%) - while that shows good improvement, it would be helpful to know how the original policy was chosen, as a 50% collision probability seems very high. I.e. in this case, could the authors provide evidence that a good selection of hand picked control parameters cannot do better than 50% coll prob to start? How much does the choice of alpha in Line 245 impact the tightness of the bound? While closed form, that does not appear to be the best choice.Clarity, Section 2.1 seems quite separate from the rest of the paper, and given the level of specificity given in the discrete time dynamics, the concepts are not well connected to what follows. Line 105 talks about "an equivalent stochastic optimization" - are these problems really equivalent? If so, equivalent in what sense, and if not, why discuss the discrete form?Also, the stochastic problemon Lines 107 and 110 appear similar to chance constrained control (an MPC) - thus the authors should discuss those similarities. Numerous papers are available - the authors could start here (though many newer results exist):@article{schwarm1999chance, title={Chance-constrained model predictive control},author={Schwarm, Alexander T and Nikolaou, Michael},journal={AIChE Journal}, volume={45}, number={8}, pages={1743--1752},year={1999} }The figure text is far too small to be readable Legends of figs 2(b), 2(c), 3(b), 3(c) have 3 lines, but the blue one is not observable  The discussion of the results at the end of section 4 could be greatly improved - there is very little discussion of the plots in the following pages, and what is there doesn't shed much more light on the results. The graphs suggest that something interesting is going on, but exactly what is hard to know.  Originality Authors point out that obtaining bounds for the cost is challenging because of the unbounded change-of-measure likelihood ratio, making approaches based on hoeffdings inequality impractical. They propose an alternative from Catoni [25], but that leads to statements like "The key step is to use the the [sic] moment truncation map \psi proposed by Catoni [25]..." [Line 232]. It is good to build on prior work, but this statement somewhat reduces the apparent originality of this work as it is indicates that Catoni's contribution was significant.SignificanceThe result is significant in that the bounds provided will enable the development of algorithms that yield policies that are robust in the sense that the resulting cost will not exceed an apriori known cost (at least with high probability).Minor points:[7] seems like an odd choice to reference for robust MPC - there better papers to consider, e.g. http://homes.esat.kuleuven.be/~maapc/static/files/CACSD/survey-robust-mpc.pdf