lucab:

If in validation, batch norm uses the wrong stats, that could explain the drop in performance.


I wouldn’t say it’s using the wrong stats, but the batch statistics will be used, if track_running_stats=True is set.



 lucab:

However, I am more puzzled by why using a loader (num_workers=1) gives different results wrt to not use it; and, by why using multiple loading processes (num_workers>1) gives different values at each run.


If you are using 3rd party libs in the data loading pipeline, you might need to seed them in the worker_init_fn. Since the fork method is used for multiprocessing on Linux, the random states might be cloned from the parent process.