ELF updates look synchronous to me. When used in this context it mean using the same clock so the processes are actually done sequentialially.
You could use that as reference but instead of when they update parameters from queue one by one just change that to how you want the gradients accumulated then updated.
You will want to use a queque to do this synchronous
Checkout the code on this part for refs https://github.com/facebookresearch/ELF/blob/master/rlpytorch/utils.py 29