Hi to everyone,
I am currently working on a FeedForward network on the MNIST database.
It has 784 inputs, two hidden layers of 100 nodes each, and an 10-node output layer.
I use PyTorch’s Adam optimizer and CrossEntropyLoss loss function.
I started from this official PyTorch example 8 and modified my way through.
Regarding the train/test steps, the actual implementation shouldn’t be too much different from that one.
My problem is the following:
The training loss has sharp jumps at each epochs, as if the Adam optimizer resets in some way (i think). Below a plot of what happens:

bokeh_plot(4).png750×330 35.3 KB

I am not uploading any of my code yet, as my implementation is quite complex and I am working with some “unusual” constraints (hopefully not relevant to the problem at hand).
However, if someone is interested, I will work toward a minimal working example.
Any help will be certainly appreciated!
Thanks,
Davide