An optimality condition in optimal control theory
The Hamilton-Jacobi-Bellman (HJB) equation is a nonlinear partial differential equation that provides necessary and sufficient conditions for optimality of a control with respect to a loss function. Its solution is the value function of the optimal control problem which, once known, can be used to obtain the optimal control by taking the maximizer (or minimizer) of the Hamiltonian involved in the HJB equation.
The equation is a result of the theory of dynamic programming which was pioneered in the 1950s by Richard Bellman and coworkers. The connection to the Hamilton–Jacobi equation from classical physics was first drawn by Rudolf Kálmán. In discrete-time problems, the analogous difference equation is usually referred to as the Bellman equation.
While classical variational problems, such as the brachistochrone problem, can be solved using the Hamilton–Jacobi–Bellman equation, the method can be applied to a broader spectrum of problems. Further it can be generalized to stochastic systems, in which case the HJB equation is a second-order elliptic partial differential equation. A major drawback, however, is that the HJB equation admits classical solutions only for a sufficiently smooth value function, which is not guaranteed in most situations. Instead, the notion of a viscosity solution is required, in which conventional derivatives are replaced by (set-valued) subderivatives.
Optimal Control Problems.
Consider the following problem in deterministic optimal control over the time period formula_1:
formula_2
where formula_3 is the scalar cost rate function and formula_4 is a function that gives the bequest value at the final state, formula_5 is the system state vector, formula_6 is assumed given, and formula_7 for formula_8 is the control vector that we are trying to find. Thus, formula_9 is the value function.
The system must also be subject to
formula_10
where formula_11 gives the vector determining physical evolution of the state vector over time.
The Partial Differential Equation.
For this simple system, the Hamilton–Jacobi–Bellman partial differential equation is
formula_12
subject to the terminal condition
formula_13
As before, the unknown scalar function formula_9 in the above partial differential equation is the Bellman value function, which represents the cost incurred from starting in state formula_15 at time formula_16 and controlling the system optimally from then until time formula_17.
Deriving the Equation.
Intuitively, the HJB equation can be derived as follows. If formula_18 is the optimal cost-to-go function (also called the 'value function'), then by Richard Bellman's principle of optimality, going from time "t" to "t" + "dt", we have
formula_19
Note that the Taylor expansion of the first term on the right-hand side is
formula_20
where formula_21 denotes the terms in the Taylor expansion of higher order than one in little-"o" notation. Then if we subtract formula_18 from both sides, divide by "dt", and take the limit as "dt" approaches zero, we obtain the HJB equation defined above.
Solving the Equation.
The HJB equation is usually solved backwards in time, starting from formula_23 and ending at formula_24.
When solved over the whole of state space and formula_25 is continuously differentiable, the HJB equation is a necessary and sufficient condition for an optimum when the terminal state is unconstrained. If we can solve for formula_26 then we can find from it a control formula_27 that achieves the minimum cost.
In general case, the HJB equation does not have a classical (smooth) solution. Several notions of generalized solutions have been developed to cover such situations, including viscosity solution (Pierre-Louis Lions and Michael Crandall), minimax solution (Andrei Izmailovich Subbotin), and others.
Approximate dynamic programming has been introduced by D. P. Bertsekas and J. N. Tsitsiklis with the use of artificial neural networks (multilayer perceptrons) for approximating the Bellman function in general. This is an effective mitigation strategy for reducing the impact of dimensionality by replacing the memorization of the complete function mapping for the whole space domain with the memorization of the sole neural network parameters. In particular, for continuous-time systems, an approximate dynamic programming approach that combines both policy iterations with neural networks was introduced. In discrete-time, an approach to solve the HJB equation combining value iterations and neural networks was introduced.
Alternatively, it has been shown that sum-of-squares optimization can yield an approximate polynomial solution to the Hamilton–Jacobi–Bellman equation arbitrarily well with respect to the formula_28 norm.
Extension to Stochastic Problems.
The idea of solving a control problem by applying Bellman's principle of optimality and then working out backwards in time an optimizing strategy can be generalized to stochastic control problems. Consider similar as above
formula_29
now with formula_30 the stochastic process to optimize and formula_31 the steering. By first using Bellman and then expanding formula_32 with Itô's rule, one finds the stochastic HJB equation
formula_33
where formula_34 represents the stochastic differentiation operator, and subject to the terminal condition
formula_35
Note that the randomness has disappeared. In this case a solution formula_36 of the latter does not necessarily solve the primal problem, it is a candidate only and a further verifying argument is required. This technique is widely used in Financial Mathematics to determine optimal investment strategies in the market (see for example Merton's portfolio problem).
Application to LQG-Control.
As an example, we can look at a system with linear stochastic dynamics and quadratic cost. If the system dynamics is given by
formula_37
and the cost accumulates at rate formula_38, the HJB equation is given by
formula_39
with optimal action given by
formula_40
Assuming a quadratic form for the value function, we obtain the usual Riccati equation for the Hessian of the value function as is usual for Linear-quadratic-Gaussian control.
References.
 ;templatestyles src="Reflist/styles.css" / ;