I have a pretty big network which can be present as Resnet+Test network. Previously I was only training the test network and always got reproducible results by fixing seeds and other things. By reproducible I mean that at every epoch of every run the result was always the same as the same epoch of the previous run.
Now I was trying to fine-tune the whole network. It means that the resnet is in the training loop. My results are not exactly reproducible anymore. Each epoch differs by a small margin. For example, among 3 different runs the Average precision(AP) after 1st epoch were: 25.101604, 25.100491,  25.095660.
I matched initial models between these runs before starting any training, they are exactly the same, also I made sure in every run they are getting the same samples.
I know the differences are not that much but I am really interested to know the reason. Any idea from anyone?