Thanks Peter.
I have updated my post with a typical use case scenario (also how I use it).
Indeed, based on my preliminary experiments, it is not possible to get the same embeddings with track_running_stats=False. Now, trying to find a solution with track_running_stats=True.
The distances between embeddings of the same images with different batch sizes are summarized below.
I wish, it was possible to also save the running mean/std of batchnorm layers from training and use them at test time with a small momentum. This would mitigate the problem.
A workaround might be to run the model on the training set with large batch size, right before switching to test. I have not tried this yet.
Do you have a better idea?