I trained a DenseNet to compute image embeddings of size 256, to match images.
At test time, when I use different batch sizes (64, 256, 512), the network outputs (embedding) are slightly different (I do model.eval()).
The difference seems to be proportional to the difference in the batch size.
I suspect, this is due to batchnorm (track_running_stats).
Update: Indeed, it is related to batchnorm’s track_running_stats.
If I set track_running_stats to False, but use a large batch size (512), it works well (high matching accuracy). When I use small batch size (8), the accuracy drops a few percent.
If I leave track_running_stats as True (default), even though the outputs for different batch sizes differ slightly, the matching accuracy is consistent across batch sizes, and slightly lower than track_running_stats = False with large batch size.
How to solve this?
I should get the same output regardless of the batch size.
Is it possible with a network with batchnorm?
PyTorch v1.4
Update 2: Typical Use case Scenario:

I train a batchnorm CNN network with a single embedding layer of size 256 with metric loss, and large batch size (256).
I also validate/test using a large batch size (512)
Got a good model, let me deploy it. Compute the embeddings for millions of images in the database using a large batch size (512). Great!
At test time, though, the network receives a single image at a time, in real-time! Oooops! The batch size is now 1. Even if I feed one of the images from the database, I do not get the same embeddings! If I match them, I do not get a distance of 0! (We cannot explain this to an outsider).
Hence, the problem.