Problem solved! Thanks to the code :https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py 3
I added     torch.manual_seed(params.seed + rank) in each of the worker to guarantee the variation between different workers and the reproduction of my algorithm.