I have already implemented A3C in pytorch, and it works just fine.  When you get a copy with everything shared in the subprocess just do this to break the gradient sharing, and use the optimizer as you’d normaly do:
for param in model.parameters():
    param.grad.data = param.grad.data.clone()

This is also covered in the notes 288, that I encourage you to read.