Basis Refinement Strategies for Linear Value Function Approximation in MDPs
We provide a theoretical framework for analyzing basis function construction for linear value function approximation in Markov Decision Processes (MDPs). We show that important existing methods, such as Krylov bases and Bellman-errorbased methods are a special case of the general framework we develop. We provide a general algorithmic framework for computing basis function refinements which "respect" the dynamics of the environment, and we derive approximation error bounds that apply for any algorithm respecting this general framework. We also show how, using ideas related to bisimulation metrics, one can translate basis refinement into a process of finding "prototypes" that are diverse enough to represent the given MDP.
This paper deals with basis construction for VFA, based on the concepts of equivalence relations and bisimulation metrics. The authors define a specific equivalence relation (consistency resp. to a subspace) and a concept of refinement and show (Th. 3.1) that starting from a reward equivalence relation, a sequence of refined bases (such that the cardinality of the newly generated basis is equal to the cardinality of the quotient set corresponding to the equivalence relation based on the span of the previous basis) will provide a (strong) guarantee regarding the quality of the optimal value function associated to this base. Then, they relax the introduced equivalence relation to take into account locality, and show a property on the link between local and global refinements. Next, they discuss a convergence criterion for this refinement principle based on bissimulation (equivalence and metric). Lastly, they extend the proposed refinement strategy by alternating metric learning and base learning (through prototypes).The following detailed comments follow the paper (they are not ordered by importance). * l.81: s_i,a_i -> s_i,r_i ? * l.86: it may be useful to define the concepts of equivalence relation, equivalence class and quotient set (as they are a core of the paper) * l.116: s1 sim s2 -> s sim s' * l.116: precise that Pa(s)(C) is the measure of the set C * l.122: a missing absolute value in the def. of Omega(d)? * l.132: what is V^*_Delta? The least-squares fixed-point approximation (for the subspace Delta(sim)) of the optimal policy pi_*? How can this be computed? * l.157: beta is not defined and the notation Gamma is used afterwards * l.161-165: it is not clear how the refining fix inconsistencies in the dynamics (eg, if f(s) neq f(s'), take varphi = f, the subspace does not change) * l.173: it should be said that sim Sigma is sim span(Sigma) (even if rather clear from the context) * Th.3.1: how could be constructed the refinement? (the authors state that sec. 5 provides an answer, but see latter comments) Given the condition on the size of the refinement, how will the sequence of subspaces grows? Isn't it possible that a few refinements could lead to the whole space F_S (in which case the result is trivial)? * l.209: what is the (pseudo)-metric 0? the metric that associates 0 to any pair of states? * l.215: the notation P^a(s)|C is not defined (consequently, the rest of the proof is obscure) * l.231: where does the inequality in Eq. 3 come from? what is the supremum norm for the norm (entrywise, induced, other)? * l.233: what is V^*_n? is it V^*_Gamma_n? * l.235: in Eq.4, gamma/(1-gamma)^2 -> 1/(1-gamma)^2 ? * Prop.3.1: how is initialized the sequence? Above all, what is Gamma_(0,eta)? As it is not defined, it is hard to understand the stated result. * Sec.3.2: it is hard to see how Krylov bases/state aggregation relate to the stated result (apart from the addressed problem, that is basis construction). Do they satisfy the assumptions of Th.3.1? * l.314: Section 3.1 -> Section 3.2 * l.330: P and M are not defined * l.389: what is a supremum over metrics? * Def.4.1: it is hard to see the implications of this def., especially how it relates to a convergence criterion for subspaces refinement * Sec.5: the proposed approach is not clear at all. Providing a pseudo-code might help (and maybe instantiating the general approach too; for example how to find a cover over S, covering with what?). * l.367: what S subset Q means? why assuming it? * l.374: hat(J)n is not defined * l.376: iota(kappa) should be iota(kappa)(s). By the way, the metric D(Gamma) takes too states as argument, how can it takes a prototype kappa as the first argument? * l.423: significantly chapter?