Epistemic Uncertainty and Observation Noise with the Neural Tangent Kernel
<|reference_start|>Epistemic Uncertainty and Observation Noise with the Neural Tangent Kernel: Recent work has shown that training wide neural networks with gradient descent is formally equivalent to computing the mean of the posterior distribution in a Gaussian Process (GP) with the Neural Tangent Kernel (NTK) as the prior covariance and zero aleatoric noise \parencite{jacot2018neural}. In this paper, we extend this framework in two ways. First, we show how to deal with non-zero aleatoric noise. Second, we derive an estimator for the posterior covariance, giving us a handle on epistemic uncertainty. Our proposed approach integrates seamlessly with standard training pipelines, as it involves training a small number of additional predictors using gradient descent on a mean squared error loss. We demonstrate the proof-of-concept of our method through empirical evaluation on synthetic regression.<|reference_end|>
@article{calvo-ordoñez2024epistemic,
    title={Epistemic Uncertainty and Observation Noise with the Neural Tangent
  Kernel},
    author={Sergio Calvo-Ordo~nez, Konstantina Palla, Kamil Ciosek},
    journal={arXiv preprint arXiv:2409.03953},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.03953},
    primaryClass={cs.LG stat.ML}
}