  Stochastic time-varying optimization is an integral part of learning in which
the shape of the function changes over time in a non-deterministic manner. This
paper considers multiple models of stochastic time variation and analyzes the
corresponding notion of hitting time for each model, i.e., the period after
which optimizing the stochastic time-varying function reveals informative
statistics on the optimization of the target function. The studied models of
time variation are motivated by adversarial attacks on the computation of value
iteration in Markov decision processes. In this application, the hitting time
quantifies the extent that the computation is robust to adversarial
disturbance. We develop upper bounds on the hitting time by analyzing the
contraction-expansion transformation appeared in the time-variation models. We
prove that the hitting time of the value function in the value iteration with a
probabilistic contraction-expansion transformation is logarithmic in terms of
the inverse of a desired precision. In addition, the hitting time is analyzed
for optimization of unknown continuous or discrete time-varying functions whose
noisy evaluations are revealed over time. The upper bound for a continuous
function is super-quadratic (but sub-cubic) in terms of the inverse of a
desired precision and the upper bound for a discrete function is logarithmic in
terms of the cardinality of the function domain. Improved bounds for convex
functions are obtained and we show that such functions are learned faster than
non-convex functions. Finally, we study a time-varying linear model with
additive noise, where hitting time is bounded with the notion of shape
dominance.

Optimization and Control