A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD
<|reference_start|>A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD: This paper addresses the issue of policy evaluation in Markov Decision Processes, using linear function approximation. It provides a unified view of algorithms such as TD(lambda), LSTD(lambda), iLSTD, residual-gradient TD. It is asserted that they all consist in minimizing a gradient function and differ by the form of this function and their means of minimizing it. Two new schemes are introduced in that framework: Full-gradient TD which uses a generalization of the principle introduced in iLSTD, and EGD TD, which reduces the gradient by successive equi-gradient descents. These three algorithms form a new intermediate family with the interesting property of making much better use of the samples than TD while keeping a gradient descent scheme, which is useful for complexity issues and optimistic policy iteration.<|reference_end|>
@article{loth2006a,
    title={A Unified View of TD Algorithms; Introducing Full-Gradient TD and
  Equi-Gradient Descent TD},
    author={Manuel Loth (INRIA Futurs), Philippe Preux (INRIA Futurs)},
    journal={Dans European Symposium on Artificial Neural Networks (2006)},
    year={2006},
    archivePrefix={arXiv},
    eprint={cs/0611145},
    primaryClass={cs.LG}
}