Gaussian Processes, not quite for dummies
Implementing Gaussian Processes in Python is a fun way to practice your programming and mathematical abilities.<p>I was working on a course project that used GPs for regression, with the inputs taken from the output of a CNN. We were trying to predict crop yield from satellite imagery, and GPs helped to incorporate spatio-temporal locality: think of predictions for neighboring county, or for image from a previous year.<p>One issue we ran into while using the model from the original authors [1] was a memory constraint from building on top of Google Colab instead of a private AWS instance, and using a larger dataset to predict upon.<p>The Gaussian kernel matrix grows O(n^2) per data point, and we had up to 22,000 datapoints to use. To get around the memory constraint, we had to re-implement the GP from scratch - carefully avoiding memory spikes that came with computing the kernel and mean predictions.<p>It was a two-fold process. For understanding the exact equations, I would highly recommend the resource mentioned in the article [2], Gaussian Processes for Machine Learning Chapter 2. For a lecture video, I would recommend [3] which is from a ML course at Cornell.<p>As for implementing GPs efficiently, this is still a challenge: numpy has a bunch of matrix-vector functions that are helpful, but there isn't a way to exploit the symmetry of the kernel matrix and use half the require memory.<p>[1] <a href="https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14435/14067" rel="nofollow">https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14435/...</a><p>[2] <a href="http://www.gaussianprocess.org/gpml/chapters/RW.pdf" rel="nofollow">http://www.gaussianprocess.org/gpml/chapters/RW.pdf</a><p>[3] <a href="https://www.youtube.com/watch?v=R-NUdqxKjos" rel="nofollow">https://www.youtube.com/watch?v=R-NUdqxKjos</a>