I got this to work and working fast if you have the GPUs:) For Pong it speed up convergence to 10mins on my dgx station compared to 45mins on CPU only.
I have posted new A3C-GPU versions in repos below:
discrete action spaces:

  
      
      GitHub
  
  
    

dgriff777/rl_a3c_pytorch 17

rl_a3c_pytorch - Reinforcement learning A3C LSTM Atari with Pytorch


  
  
    
    
  
  


continuous action space version:

  
      
      GitHub
  
  
    

dgriff777/a3c_continuous 7

a3c_continuous - A continuous action space version of A3C LSTM in pytorch