  Reinforcement learning based adaptive/approximate dynamic programming (ADP)
is a powerful technique to determine an approximate optimal controller for a
dynamical system. These methods bypass the need to analytically solve the
nonlinear Hamilton-Jacobi-Bellman equation, whose solution is often to
difficult to determine but is needed to determine the optimal control policy.
ADP methods usually employ a policy iteration algorithm that evaluates and
improves a value function at every step to find the optimal control policy.
Previous works in ADP have been lacking a stronger condition that ensures the
convergence of the policy iteration algorithm. This paper provides a sufficient
but not necessary condition that guarantees the convergence of an ADP
algorithm. This condition may provide a more solid theoretical framework for
ADP-based control algorithm design for nonlinear dynamical systems.

Optimization and Control