In machine learning, a neural differential equation is a differential equation whose right-hand side is parametrized by the weights "θ" of a neural network. In particular, a neural ordinary differential equation (neural ODE) is an ordinary differential equation of the form
formula_1In classical neural networks, layers are arranged in a sequence indexed by natural numbers. In neural ODEs, however, layers form a continuous family indexed by positive real numbers. Specifically, the function formula_2 maps each positive index "t" to a real value, representing the state of the neural network at that layer.
Neural ODEs can be understood as continuous-time control systems, where their ability to interpolate data can be interpreted in terms of controllability.
Connection with residual neural networks.
Neural ODEs can be interpreted as a residual neural network with a continuum of layers rather than a discrete number of layers. Applying the Euler method with a unit time step to a neural ODE yields the forward propagation equation of a residual neural network:
formula_3
with ℓ being the ℓ-th layer of this residual neural network. While the forward propagation of a residual neural network is done by applying a sequence of transformations starting at the input layer, the forward propagation computation of a neural ODE is done by solving a differential equation. More precisely, the output formula_4 associated to the input formula_5 of the neural ODE is obtained by solving the initial value problem
formula_6
and assigning the value formula_7 to formula_4.
Universal differential equations.
In physics-informed contexts where additional information is known, neural ODEs can be combined with an existing first-principles model to build a physics-informed neural network model called universal differential equations (UDE). For instance, an UDE version of the Lotka-Volterra model can be written as
formula_9
where the terms formula_10 and formula_11 are correction terms parametrized by neural networks.
References.
 ;templatestyles src="Reflist/styles.css" / ;