+++++++++++++++++++++++++++++++++++++++++++++++++++++++
                      PAGE 5/5

+++++++++++++++++++++++++++++++++++++++++++++++++++++++
Now back to the issue in code 3. We show in an illustrative example that random number generation is done sequentially. This implies that generating random numbers for 4 samples will provide the 4th sample with some random values that will be different IF the 4th sample was processed on another device with position 0 in the mini-batch. This prevents reproducibility across ALL K.
In this case, we see random numbers generation over CPU and GPU.
code 5: Random number generation is SEQUENTIAL whether on CPU or on GPU.
import threading
import random

import numpy as np

import torch
import torch.nn as nn
from torch.nn import DataParallel


def set_seed(seed):
    """
    For seed to some modules.
    :param seed: int. The seed.
    :return:
    """
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True


class Model(nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.thread_lock = threading.Lock()

    def forward(self, x, seed):
        if x.is_cuda:
            device = x.get_device()
        else:
            device = torch.device("cpu")

        self.thread_lock.acquire()
        print("Generating random number on CPU on DEVICE {}".format(torch.cuda.current_device()))
        for i in range(1, x.size()[0], 1):
            set_seed(seed)  # re-seed
            delta = torch.rand(i)
            print("Random array {} at DEVICE {}: {}".format(i, torch.cuda.current_device(), delta))
            set_seed(seed)  # re-seed like nothing happened!!!

        print("Generating random number on GPU on DEVICE {}".format(torch.cuda.current_device()))
        for i in range(1, x.size()[0], 1):
            set_seed(seed)  # re-seed
            shape = torch.Size((i, 1))
            delta = torch.cuda.FloatTensor(shape)
            torch.rand(shape, out=delta)
            print("Random array {} at DEVICE {}: {}".format(i, torch.cuda.current_device(), delta.view(-1)))
            set_seed(seed)  # re-seed like nothing happened!!!
        self.thread_lock.release()

        return x


if __name__ == "__main__":
    set_seed(0)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    model = Model()
    model = DataParallel(model)
    model.to(device)

    x = torch.rand(8, 3)
    x = x.to(device)
    model(x=x, seed=torch.tensor([10, 20]).to(device))


run:
Generating random number on CPU on DEVICE 0
Random array 1 at DEVICE 0: tensor([0.4581])
Random array 2 at DEVICE 0: tensor([0.4581, 0.4829])
Random array 3 at DEVICE 0: tensor([0.4581, 0.4829, 0.3125])
Generating random number on GPU on DEVICE 0
Random array 1 at DEVICE 0: tensor([0.2479], device='cuda:0')
Random array 2 at DEVICE 0: tensor([0.2479, 0.8615], device='cuda:0')
Random array 3 at DEVICE 0: tensor([0.2479, 0.8615, 0.0881], device='cuda:0')
Generating random number on CPU on DEVICE 1
Random array 1 at DEVICE 1: tensor([0.5615])
Random array 2 at DEVICE 1: tensor([0.5615, 0.1774])
Random array 3 at DEVICE 1: tensor([0.5615, 0.1774, 0.8147])
Generating random number on GPU on DEVICE 1
Random array 1 at DEVICE 1: tensor([0.8389], device='cuda:1')
Random array 2 at DEVICE 1: tensor([0.8389, 0.3764], device='cuda:1')
Random array 3 at DEVICE 1: tensor([0.8389, 0.3764, 0.6292], device='cuda:1')

You see that numbers are generated sequentially.
You may ask why the numbers are different from CPU to GPU?. I do not have an explanation. I expect to have the same numbers. MY GUESS is that it may have something to do with the sampling algorithm. They may be using different algorithm depending if the sampling is done on CPU or GPU [check THRandom.cpp 1]. Pytorch does not guarantee reproducibility between CPU and GPU.
The consequence of the previous note is that if you use for instance dropout, the mask generated for a sample depends on its location on which device and on its order in the minibatch. If we fixed the same seed all over the devices, it  will depend only on its position in the minibatch. This position will change when changing the number of devices   K. This makes reproducibility across all K impossible even using our re-seeding strategy. Now, you may say,   what if we fix a seed per SAMPLE so to be independent of any device. Conceptually, that will solve the problem.  However, in practice, this may slow down the computation since you will perform sequential processing of sample.  For instance, in the dropout, you have to forward each sample separately to be able to fix its own seed. This is  not practical at all.
Last notes/reminders:

If you want to make your code reproducible for each fixed K GPUs, you can use a seed per thread, combined with locks. You need to lock the smallest number of instructions that produce randomness. If a model produces randomness, do not lock the entire call to  the model, but get in the model, and lock only the instructions that generate randomness.
You need to deal with BatchNorm to not breakdown performance, in case BN is used by using synchronized BN across all GPUs.
Reproducibility in Pytorch still needs a lot of work.