Thank you for sharing!
It’s not technical commentary, but there are two enhancements that I’d suggest:

Also, for some people Jupyter notebooks with a convergence graph or so may make it more attractive. At least I, personally, always look for notebooks first. (Opinions may differ though.)
I’d probably try to make things more PyTorchy, e.g.:

Replace the dependency on keras for the datasets by one on torchvision,
provide a dataloader that facilitates the overlapping sampling (I must admit I’m not sure how easy that is),
consider integrating the various bits of the optimizer into optimizer.step and pass a closure.



Another thing to try might be reaching out to the GPyTorch people / include an example with Gaussian processes, I’ve always thought of (L-) BFGS as the premier way to train GPs.
Best regards
Thomas