Maybe Hogwild isn’t the right term, but here’s what I want to achieve:
Take distributed DQN as an example. Each worker does the following repeatedly:


Each has a local copy of the global parameter.


Interact with the Atari simulator, sample experience from replay memory, and compute gradient on GPU. The policy network can potentially be a big convnet, so GPU will accelerate a lot.


Send the gradient to the parameter server. The PS uses shared Adagrad or whatever to update the central parameter copy.


Pull from PS to update the local parameter copy.


This is the async GPU training I want to implement. There are many use cases outside RL too, like elastic SGD.