Thanks for your feedback!
To finish this discussion: The gradients were computed correctly and independently in the child processes. However, I was writing back results from the different child processes into the same CUDA memory location. That naturally lead to inconsistent results.