@Dawid_S try to change the learning rate a bit.
I also have the same problem.
In a while, I trust SGD more than adam,
but it doesn’t make sense when the others report that adam have better performance than SGD. And it turns out that we must set correct learning rate to start.
source: from my bloody experience using adam.
CMIIW. I also want to know why this bug happen all of the time
best regards,
Albert Christianto