A Block Coordinate Ascent Algorithm for Mean-Variance Optimization
Risk management in dynamic decision problems is a primary concern in many fields, including financial investment, autonomous driving, and healthcare. The mean-variance function is one of the most widely used objective functions in risk management due to its simplicity and interpretability. Existing algorithms for mean-variance optimization are based on multi-time-scale stochastic approximation, whose learning rate schedules are often hard to tune, and have only asymptotic convergence proof. In this paper, we develop a model-free policy search framework for mean-variance optimization with finite-sample error bound analysis (to local optima). Our starting point is a reformulation of the original mean-variance function with its Legendre-Fenchel dual, from which we propose a stochastic block coordinate ascent policy search algorithm. Both the asymptotic convergence guarantee of the last iteration's solution and the convergence rate of the randomly picked solution are provided, and their applicability is demonstrated on several benchmark domains.
The paper "A Block Coordinate Ascent Algorithm for Mean-Variance Optimization" proposes a new algorithm for reinforcement learning with a mean-variance objective that has a hard constraint on the reward variance and analyses the convergence of the algorithm in both infinite and finite horizon cases. The derivations for the new algorithm use a transformation based on a Fenchel dual (to transform a quadratic term into a linear term plus terms with an added additional variable) enabling gradient descent to be applied on the transformed Lagrangian dual of the original objective. The contribution w.r.t. the proofs and the new algorithm appear solid.  The paper is in general well written. There are some typos that should be fixed. The main problem in the paper is the reporting of the experimental results. The representation used for the policy is not discussed, the variance limit is not defined, and it is unclear whether the reported results are statistically significant.  The main claims about related work are as follows: "1) Most of the analyses of ODE-based methods are asymptotic, with no sample complexity analysis." Most are asymptotic? The paper needs to discuss those cases where the analysis is not asymptotic. This is discussed later in the paper? Should be already mentioned here. "2) It is well-known that multi-time-scale approaches are sensitive to the choice of the stepsize schedules, which is a non-trivial burden in real-world problems." "3) The ODE approach does not allow extra penalty functions. Adding penalty functions can often strengthen the robustness of the algorithm, encourages sparsity and incorporates prior knowledge into the problem [Hastie et al., 2001]."  Claims 1) - 3) seem correct and the proposed algorithm could offer some improvement over these previous approaches.  Algorithm 1: Each episode starts from the initial state s_1 until ending in the recurrent state s^*. It could help the reader to give an explanation why this is done so? Is this absolutely necessary for the analysis of the algorithm?  EXPERIMENTS: In the experiments, it is unclear what kind of representation the policy follows. It cannot be tabular since some of the problems have continuous states. It is also unclear whether the form of the policy influences the performance of the algorithms.  \xi, the limit on the expected variance of the rewards (Equation 1), should be defined for each problem.  The statistical significance of the results is now described only in textual form. For example, "The results indicate that MVP yields a higher mean return with less variance compared to the competing algorithms."  Firstly, according to my understanding the objective is not to minimize variance but to limit it, at least according to Equation 1. As such the variance of the comparison methods does not matter if it remains under the limit (which is not defined). Of course the variance is of separate interest for understanding how far each algorithm is from the limit. I recommend reporting whether the variance limit is exceeded and by how much. Furthermore, I recommend performing statistical significance testing on both the mean and variance.  LANGUAGE: In some places there is some unclarity that should be fixed. For example, in "yet almost all the reported analysis of these algorithms are asymptotic [Xu and Yin, 2015]." almost all? Which reported analysis is not asymptotic?  "In risk-sensitive mean-variance optimization MDPs, the objective is often to maximize J(θ) with a variance constraint, i.e.," *often* ? Earlier in the paper there is a discussion on related work but it was not made clear what exactly the objective in those papers is?  Smaller typos: Line 85: "the third time-scale is to optimizes over" -> the third time-scale optimizes over Line 48: "2 Backgrounds" -> "2 Background" ? Line 116: "and derives novel algorithms" -> and derive novel algorithms Line 188: "for which there is no established results" -> for which there are no established results Line 203: "Without the loss of generality" should be usually "Without loss of generality" Line 205: "as as" -> as Line 206: "approximate error" -> approximation error Line 219: "Now it is ready to provide" -> Now we provide