Yes, I do shuffle the data, but with replacement… I’m doing a word2vec kind of thing, so I drawing with probabilities that varies with the number of times a certain word appears in my dataset. So the distribution must always be the same.
I’ve just run two trainings, saving all the parameters from the optimizer (the state dict and learning rate) and i got this: