Hi @apaszke thanks for the code!
As you mentioned your private impl of A3C, I have one more simple question: according to the paper, A3C only synchronizes local network parameters periodically with the shared network parameters (in contrast to Asynchronous N-step Q where the network parameters is always synchronized).  To do this, did you write code like:
# after every several steps (e.g., 5 or 20)
for t_param, shared_param in zip(t_model.parameters(), shared_model.parameters()):
    t_param.data.copy_(shared_param.data)

Or did you find it not critical to the accuracy in your implementation? (Previously, I strictly follow the paper and I can reproduce the scoring curve for breakout as in the paper’s figure)