Bellman Error Based Feature Generation using Random Projections on Sparse Spaces
This paper addresses the problem of automatic generation of features for value function approximation in reinforcement learning. Bellman Error Basis Functions (BEBFs) have been shown to improve policy evaluation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algorithm based on random projections, which generates BEBFs for sparse feature spaces. We provide a finite sample analysis of the proposed method, and prove that projections logarithmic in the dimension of the original space guarantee a contraction in the error. Empirical results demonstrate the strength of this method in domains in which choosing a good state representation is challenging.
This paper proposes to use random projections as a 
      proxy to learn BEBFs (Bellman Error Basis Functions). Given a (high 
      dimensional) set of features and the currently estimated value function, 
      the features are (randomly) projected on a smaller space, and the temporal 
      differences errors (related to the currently estimated value function) are 
      regressed on these projected features. The (scalar) regressed function is 
      then added to the set of features used to estimate the value function. A 
      finite sample analysis is conducted, the main result showing that if the 
      Bellman residual is linear in the (high dimensional) features, then the 
      Bellman error can be well regressed on the compressed space (depending 
      notably on the size of this space and on the number of samples). The 
      authors also use this result to provide some guarantee on the estimated 
      value function. The proposed algorithm is compared to state-of-the-art 
      approaches on a high dimensional problem. This paper is very 
      clearly written and presents some quite interesting contributions, summed 
      up above. I have mainly two comments. 1) First, I found that lemma 4 
      was not so clear, which is quite a pity as it summarizes the preceding 
      results and quantifies the efficiency of the proposed approach. Notably, 
      how strong is the assumption of the existence of constants c1 to c3? Also, 
      is it straightforward that the universal (?) constants c4 and c5 exist (or 
      is this an assumption?). Considering the provided bound, we should get 
      $\gamma_0$ as small as possible, which seems actually a good thing also 
      for the choice of $d$ and $n$... this seems quite surprising. What did I 
      miss? I think that a proof (in the appendix) may help (even if considered 
      as straightforward) 2) Second, I have some comments regarding the 
      experimental section: * what is the reward? * it would be 
      interesting to consider also CBEBF (the not simple version) in the 
      experiments, even if it has a higher computational cost, to see what can 
      be gained in estimation quality (above l2-LSTD?) * the results of 
      l2-LSTD are quite surprising... a validation set is mentioned, but 
      cross-validation for value estimation does not seem to be straightforward. 
      Do you use the NRPE to cross-validate? * it would have been 
      interesting to compare your algorithm to iLSTD [1], which pursue a similar 
      goal (sample efficiency and low computational cost) * why do you 
      change the schedule for dimensions between experiments (as it is the same 
      problem)? * what is the $\mu$ distribution in the experiment (used for 
      NRPE)? * that is true that the gradient-based approaches should be 
      less sample-efficient (and implies the choice of some possibly sensitive 
      meta-parameters), but it would have been intersting to show it 
      (empirically) * regarding the last experiment, it would have been 
      interesting to see if the efficiency of l2-LSTD degrades when the feature 
      are not sparse (e.g. features transformed through a linear mapping). This 
      is actually mentioned in the text (no change), but fig.2 suggests that the 
      feature vector is quite sparse (400 non-zero over 65536). [1] 
      Alborz Geramifard, Michael Bowling, and Richard S. Sutton. Incremental 
      Least-Squares Temporal Difference Learning. In AAAI. 
      2006