The problem does not exist any more.
I did not “control variable” when compare the time cost between with/without synchronize. So sth like CPU load, samples order etc may affect the result.
When I run code with/without CUDA_LAUNCH_BLOCKING=1 at the same time with the same random seed, the time cost are different.
Anyway, thank you very much.