Here are two graphs of training my model with only 1 data sample. SGD is perfectly monotonic, while Adam has these weird oscillations. Why is this?
After creating the model, I saved model.state_dict(), trained with SGD, then reloaded the original state dict, and finally trained with Adam.
image1417×1120 57.4 KB image1553×1305 82.1 KB
For reference, here’s 100 samples (batch_size = 1):
image1795×1343 119 KB
image1516×1241 150 KB
Prompted by this: Training with batch_size = 1, all outputs are the same and trains poorly