ELF (rlpytorch) provides source code for synchronous updates:

  
      
      GitHub
  
  
    

facebookresearch/ELF 70

An End-To-End, Lightweight and Flexible Platform for Game Research - facebookresearch/ELF