Just a quick follow-up for those wanting to implement A3C.
A3C uses a Hogwild! update style, which means updates are made lock-free. Workers possibly overwrite each other updates at times, but that’s OK.
The only thing needed for Adam or RMSprop to be shared with other processes is override the classes and call the share_memory method on the important variables. One can also cleverly work around the step counter which is not a tensor, so it can’t be shared.
But other than that, it is doable, and not too hard.