Each process should execute the seeding code.
The slowdown might come from e.g. cudnn.deterministic = True, as this will use the default algorithm, which might be slower than the others.
Also, cudnn.benchmark = True won’t have any effect, if you set cudnn.deterministic = True.