Overall Simulated Annealing worked well enough to get 72% test set accuracy on my dataset, which was about the same as SGD. But with Adam I got to over 90% on that dataset. When I get some free time I can post a short write up with code and show training curves for different hyperparameters on Simulated Annealing