@ptrblck
Thanks a lot, but do not be affraid I am more concerned by systematics and I will not pretend to get better model than others. Now, all of you point that it is a sign of instability. Good, as I was tracking a bug  Let me more elaborate the use case: I’m using an Inception like model and try to train it with Adam with eps=0.1. I’m using torch but in tensorflow doc for this minimizer it is explicitely stated that for Inception on should use eps=0.1 or 1 and not using the 10^-8 default value. By the way I have experimented with eps=0.01 and the results were completly instable.
May be you have nice advise: eg. an other minimizer (eg. SGD)…? notice that I have used also AdamW and concerning the seed behavior I have not got more stable seed free results.