  We introduce the receding-horizon policy gradient (RHPG) algorithm, the first
PG algorithm with provable global convergence in learning the optimal linear
estimator designs, i.e., the Kalman filter (KF). Notably, the RHPG algorithm
does not require any prior knowledge of the system for initialization and does
not require the target system to be open-loop stable. The key of RHPG is that
we integrate vanilla PG (or any other policy search directions) into a dynamic
programming outer loop, which iteratively decomposes the infinite-horizon KF
problem that is constrained and non-convex in the policy parameter into a
sequence of static estimation problems that are unconstrained and
strongly-convex, thus enabling global convergence. We further provide
fine-grained analyses of the optimization landscape under RHPG and detail the
convergence and sample complexity guarantees of the algorithm. This work serves
as an initial attempt to develop reinforcement learning algorithms specifically
for control applications with performance guarantees by utilizing classic
control theory in both algorithmic design and theoretical analyses. Lastly, we
validate our theories by deploying the RHPG algorithm to learn the Kalman
filter design of a large-scale convection-diffusion model. We open-source the
code repository at \url{https://github.com/xiangyuan-zhang/LearningKF}.

Optimization and Control;Artificial Intelligence;Machine Learning;Systems and Control;Systems and Control;Dynamical Systems