This paper addresses the problem of automatic generation of features for value function approximation in reinforcement learning. Bellman Error Basis Functions (BEBFs) have been shown to improve policy evaluation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algorithm based on random projections, which generates BEBFs for sparse feature spaces. We provide a finite sample analysis of the proposed method, and prove that projections logarithmic in the dimension of the original space guarantee a contraction in the error. Empirical results demonstrate the strength of this method in domains in which choosing a good state representation is challenging.
['Mahdi Milani Fard', 'Yuri Grinberg', 'Amir Massoud Farahmand', 'Joelle Pineau', 'Doina Precup']
Bellman Error Based Feature Generation using Random Projections on Sparse Spaces