Hello, David. This is a really clear and concise implementation of DPPO! Only a question, how do you guarantee the variation in actions between different workers. In my game environment, every worker faces the same initial state, and I use torch.distributions.sample function to sample actions in worker. Unfortunately, I found that the continuous actions sampled by different workers are exactly the same! I use
np.random.seed(params.seed)
torch.manual_seed(params.seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
to guarantee the reproducibility of the algorithm. So I think it might have some influence on the variation between workers. And I wonder how you solve it.