Hi Frank !
It worked ! With the replacement method there are no more drops.  Thank you so much.
Concerning your 100 runs graph, I’ll stop at 50 (it takes too much time). There isn’t much difference with 25, just a little bit less noise and more defined drops.
So it seems that the problem was coming from the DataLoader. I’d still like to know why, as I think that my first approach should have worked… I did a quick test, following the Pytorch tutorial 2 (I slightly modified it to get batches of size 32, more epochs, and all the losses) I trained 10 small models. The drops are already visible.

Unknown1173×860 66.6 KB

Is there any known issue with the DataLoader or the Sampler? Maybe a statistical bias or something like that?
Thanks,
William