Asynchronous parameters updating?
<SCODE>-- in main thread: shared parameters
params, _ = sharedNet:getParameters()

-- in worker thread: its own gradParameters
tNet = sharedNet:clone()
_, gradParams = tNet:getParameters()

-- in worker thread: stuff

-- in worker thread: updating shared parameters with its own gradParameters
function feval() return nil, gradParams end
optim.rmsprop(feval, params, sharedStates)
<ECODE> Thanks in advance!
pengsun
I have already implemented A3C in pytorch, and it works just fine.  When you get a copy with everything shared in the subprocess just do this to break the gradient sharing, and use the optimizer as you’d normaly do: <SCODE>for param in model.parameters():
    param.grad.data = param.grad.data.clone()
<ECODE>
apaszke
<SCODE># after every several steps (e.g., 5 or 20)
for t_param, shared_param in zip(t_model.parameters(), shared_model.parameters()):
    t_param.data.copy_(shared_param.data)
<ECODE> Or did you find it not critical to the accuracy in your implementation? (Previously, I strictly follow the paper and I can reproduce the scoring curve for breakout as in the paper’s figure)
pengsun
<SCODE>t_model.load_state_dict(shared_state_dict)
<ECODE>
apaszke
pengsun
Anurup_Arvind
apaszke
It converges on PongDeterministic-v3 in 10 minutes with 16 threads. However, it work poorly on PongDeterministic-v3. Could you please take a look? I wonder whether I got asynchronous updates in PyTorch right.
Ilya_Kostrikov
I don’t think I’d recommend reusing the same grad tensors in multiple Variables, but apart form that I can’t see anything wrong at a glance.
apaszke
The problem was in the architecture that I used initially, seems to work on Breakout now. Reusing the same grad tensor might cause some problems?
Ilya_Kostrikov
apaszke
AttributeError: ‘float’ object has no attribute ‘backward’ Any guesses what’s caused it? Apart from my custom environment, the only difference to your code is I’ve remove the conv layers, and used a 2 layer LSTM - which is the model I’m using in TensorFlow. As a performance improvement have you tried concatenating the previous action, reward, and a timestep counter onto the end of the state as 3 scalars - I noticed a significant improvement in my TF implementation when I do this.
AjayTalati
apaszke
Which I’m working through, which seems closer to my TF implementation. Hopefully I’ll figure it out soon. At the moment my environment code, (I guess like yours), is stuck in the middle of a larger project, and is not stand alone, yet. This is actually the reason why I’m moving away from TF. Hopefully, I’ll be able to wrap my environment into something like an OpenAI env, and be able to open source it on Github within a few days, and get back to you. It’s a nice example of graph optimisation/travelling salesman, so it should be of interest to quite a few people. All the best, Aj
AjayTalati
About Variables, you only need to use the for parts that will require differentiation, and it’s best to not use them for everything else. If you have any questions about any specific cases, I’ll be happy to answer.
apaszke
you mean right?
onlytailei
denizs
Hung_Nguyen