Uncertainty Estimation of Deep Neural Networks
Normal neural networks trained with gradient descent and back propagation have received great success in various applications. On one hand, point estimation of the network weights is prone to over fitting problems and lacks important uncertainty information associated with the estimation. On the other hand, exact Bayesian neural network methods are intractable and non applicable for real world applications. To date, approximate methods have been actively under development for Bayesian neural networks, including but not limited to: stochastic variational methods, Monte Carlo dropouts, and expectation propagation. Though these methods are applicable for current large networks, there are limits to these approaches with either underestimation or over estimation of uncertainty. Extended Kalman filters (EKFs) and unscented Kalman filters (UKFs) which are widely used in data assimilation community, adopt a different perspective of inferring the parameters. Nevertheless, EKFs are incapable of dealing with highly non linearity, while UKFs are inapplicable for large network architectures. Ensemble Kalman filters (EnKFs) serve as great methodology in atmosphere and oceanology disciplines targeting extremely high dimensional, non Gaussian, and nonlinear state space models. So far, there is little work that applies EnKFs to estimate the parameters of deep neural networks. By considering neural network as a nonlinear function, we augment the network prediction with parameters as new states and adapt the state space model to update the parameters. In the first work, we describe the ensemble Kalman filter, two proposed training schemes for training both fully connected and Long Short term Memory (LSTM) networks, and experiment