Hi Pieter.
Thanks for your answer. This looks odd indeed. There is no contamination between the sets. Everything is in separated folders and the dataset class only gets the right folder.
I did try many things.
Even when I set to train, never switch to eval and only do training steps without ever going to the validation dataset this still occurs at the beginning of the epochs. I tried with either torch.utils.data.distributed.DistributedSampler or simply with random sampling without splitting the dataset between machines.

I am using filesystem synchronisation.
My code is fairly based on the imagenet example.
I am using batchnorm and have seen that it does not get properly synchronized but it should not be a problem I guess since the distribution of my samples should stay the same.
I do have some custom initialisations in the __init__ of my modules.

If I find something I will keep this thread updated.