An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks
<|reference_start|>An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks: Temporal difference (TD) learning algorithms with neural network function parameterization have well-established empirical success in many practical large-scale reinforcement learning tasks. However, theoretical understanding of these algorithms remains challenging due to the nonlinearity of the action-value approximation. In this paper, we develop an improved non-asymptotic analysis of the neural TD method with a general $L$-layer neural network. New proof techniques are developed and an improved new $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity is derived. To our best knowledge, this is the first finite-time analysis of neural TD that achieves an $\tilde{\mathcal{O}}(\epsilon^{-1})$ complexity under the Markovian sampling, as opposed to the best known $\tilde{\mathcal{O}}(\epsilon^{-2})$ complexity in the existing literature.<|reference_end|>
@article{ke2024an,
    title={An Improved Finite-time Analysis of Temporal Difference Learning with
  Deep Neural Networks},
    author={Zhifa Ke, Zaiwen Wen, Junyu Zhang},
    journal={arXiv preprint arXiv:2405.04017},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04017},
    primaryClass={cs.LG cs.AI math.OC}
}