Thanks for your kind reply.
(1)Yes, i am sure that get 0.8% accuracy difference, if you run the pytorch / example code for imagenet, you will get the same truth, that difference definitely  exists, some time the difference is small  0.1%, some time you may get 0.3% or larger difference between accuracy. Difference seed and num_worker might cause different gap between the two method i am not sure for the truth.
(2)By the way, i am not able to solve the reproducible problem for pytorch with distributeddataparallel. i have try to follow the setting with cudnn.benchmark=False cudnn.deterministic=True, and set the  torch seed and cuda seed for each rank and dataloader worker_init_fn ( numpy seed and random.seed ) for dataloader and fix the num_workers  to a fixed number every time runing my code, but result always different. how can we reproduce our own experiments with the same setting.
(3)As for the accuracy, each rank save the index for error file and write to a json file. at last i summary all the json file to get the accuracy. this is an accurate method.  i have also follow the dist.all_reduce method to get a similar result. in both case, difference quite exists and some time quite be unexpected.
so, i do really think sync bn parameters is quite important for some tasks, when we can not shared a big batch size.
(4) As for the difference between our model in each rank , should we all_reduce all model for each rank  right after each epoch, so bn parameter get all_reduce for average. this may not cause too much time when we run in multi-node.