I found a possible answer. @Yanzzzzz @wwiiiii

  
    
    
    Grad sharing problem in a3c autograd
  
  
    yeah it really there just to make sure initializing goes smoothly. 
async updates without locks mean they updates from the parallel processes are done asynchronously which means they are run on their own clock and the different process threads can actually make updates at the same time which can lead to bad updates. The reason they do this even though there is risk of bad updates being made is that updates are done much faster which speeds up training as the process of acquiring and releasing lo…
  


And  I found another related answer

  
    
    
    Model update with "share_memory" need lock protection reinforcement-learning
  
  
    TL;DR. 
Maybe a lock is needed for optimizer.step() while using multiprocessing. 

The example provided in [MULTIPROCESSING BEST PRACTICES] (https://pytorch.org/docs/stable/notes/multiprocessing.html) is to update parameters without lock protection. 
import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        o…