PAGE 2


OK, I see what you did there. But, How to obtain the same results when using different number of worker k>=0:
As we saw earlier, the internal state of the main process and the workers evolve differently. If one want to control the state of the main process, we need to reseed its state every epoch as well. The main process may do many things and change its many of its modules’ internal states. So, we need to reseed all possible modules that inrroduce randomness. Here is the code:

import random
import sys

import numpy as np
import torch

from torch.utils.data.dataloader import DataLoader
from torch.utils.data import Dataset

print("Pytorch: {}".format(torch.__version__))
print("Python: {}".format(sys.version))

np.random.seed(0)

x = np.arange(30)


def _init_fn(worker_id):
    # Not necessary since we will reinitialize the internal generator state of the worker at EVERY SAMPLE!
    pass


def transform(v):
    return torch.rand(1).item()


class TestDataset(Dataset):
    def __init__(self):
        super(TestDataset, self).__init__()
        self.seeds = None
        self.set_up_new_seeds()  # set up seeds for the initialization.

    def set_up_new_seeds(self):
        self.seeds = self.get_new_seeds()

    def get_new_seeds(self):
        return np.random.randint(0, 100, len(self))

    def __len__(self):
        return 30

    def __getitem__(self, index):
        # Set the seed for this sample: Seed ALL THE MODULES within the worker that need seeding.
        # In this example: we seed only torch. If you use numpy or other modules to load your samples, you need to
        # seed them as well in this place.
        seed = self.seeds[index]
        torch.manual_seed(seed)

        return transform(index), x[index]


dataset = TestDataset()
seed = 0
torch.manual_seed(seed)  # for reproducibility for the same run.
num_workers = 4
print("num_workers {}".format(num_workers))
loader = DataLoader(dataset, batch_size=4, shuffle=True, num_workers=num_workers, drop_last=True, worker_init_fn=_init_fn)
for i in range(4):
    print("epoch {}".format(i + 1))
    iteration = 0
    # Reseed the modules that introduce randomness: for the case num_workers = 0 (the main process).
    np.random.seed(i)
    random.seed(i)
    torch.manual_seed(i)
    # Initialize the seeds before creating the workers:
    dataset.set_up_new_seeds()

    for batch in loader:
        iteration += 1
        print("batch:", iteration, batch)
    print("Seeds at epoch {}: {}".format(iteration, dataset.seeds))



k=4:
Pytorch: 1.0.0
Python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55)
[GCC 7.3.0]
num_workers 4
epoch 1
batch: 1 [tensor([0.2919, 0.5615, 0.0036, 0.6104], dtype=torch.float64), tensor([23, 26, 17, 27])]
batch: 2 [tensor([0.4736, 0.4978, 0.7380, 0.6558], dtype=torch.float64), tensor([ 7,  8,  4, 25])]
batch: 3 [tensor([0.0492, 0.7196, 0.0530, 0.6611], dtype=torch.float64), tensor([ 2,  0,  1, 18])]
batch: 4 [tensor([0.7518, 0.9873, 0.7731, 0.7731], dtype=torch.float64), tensor([22, 20, 12, 11])]
batch: 5 [tensor([0.4657, 0.3250, 0.0036, 0.4670], dtype=torch.float64), tensor([13, 24, 16, 14])]
batch: 6 [tensor([0.7911, 0.0036, 0.7731, 0.7380], dtype=torch.float64), tensor([21,  9, 19,  3])]
batch: 7 [tensor([0.8029, 0.1710, 0.9731, 0.4350], dtype=torch.float64), tensor([ 6, 29, 15, 10])]
Seeds at epoch 7: [44 47 64 67 67  9 83 21 36 87 70 88 88 12 58 65 39 87 46 88 81 37 25 77
 72  9 20 80 69 79]
epoch 2
batch: 1 [tensor([0.0036, 0.1625, 0.8303, 0.6180], dtype=torch.float64), tensor([24,  8,  5, 22])]
batch: 2 [tensor([0.5073, 0.5695, 0.0918, 0.7865], dtype=torch.float64), tensor([11, 21, 29, 17])]
batch: 3 [tensor([0.8674, 0.5615, 0.1628, 0.1710], dtype=torch.float64), tensor([28, 15, 26,  6])]
batch: 4 [tensor([0.0036, 0.9598, 0.1226, 0.3250], dtype=torch.float64), tensor([25, 27, 20,  2])]
batch: 5 [tensor([0.6180, 0.6275, 0.7911, 0.7518], dtype=torch.float64), tensor([14, 19,  0, 13])]
batch: 6 [tensor([0.5724, 0.4657, 0.5722, 0.7576], dtype=torch.float64), tensor([10,  1, 12,  9])]
batch: 7 [tensor([0.3991, 0.0492, 0.6558, 0.1490], dtype=torch.float64), tensor([23,  7,  3, 18])]
Seeds at epoch 7: [37 12 72  9 75  5 79 64 16  1 76 71  6 25 50 20 18 84 11 28 29 14 50 68
 87 87 94 96 86 13]
epoch 3
batch: 1 [tensor([0.6487, 0.7911, 0.7380, 0.2298], dtype=torch.float64), tensor([11, 18, 20, 15])]
batch: 2 [tensor([0.8718, 0.3250, 0.8089, 0.5596], dtype=torch.float64), tensor([ 8,  2, 14, 21])]
batch: 3 [tensor([0.4540, 0.0036, 0.7731, 0.0530], dtype=torch.float64), tensor([ 4, 19, 29, 13])]
batch: 4 [tensor([0.3659, 0.0469, 0.5737, 0.5615], dtype=torch.float64), tensor([ 3,  9, 10, 17])]
batch: 5 [tensor([0.4578, 0.8157, 0.8823, 0.5349], dtype=torch.float64), tensor([ 5, 12, 22,  7])]
batch: 6 [tensor([0.9872, 0.1819, 0.6186, 0.7380], dtype=torch.float64), tensor([23, 16, 25, 27])]
batch: 7 [tensor([0.8398, 0.6487, 0.2973, 0.5019], dtype=torch.float64), tensor([28,  6,  1, 24])]
Seeds at epoch 7: [40 15 72 22 43 82 75  7 34 49 95 75 85 47 63 31 90 20 37 39 67  4 42 51
 38 33 58 67 69 88]
epoch 4
batch: 1 [tensor([0.7638, 0.4736, 0.9531, 0.2267], dtype=torch.float64), tensor([ 7, 10,  2, 15])]
batch: 2 [tensor([0.4976, 0.4736, 0.9873, 0.5019], dtype=torch.float64), tensor([22,  5, 19, 11])]
batch: 3 [tensor([0.0036, 0.3250, 0.4581, 0.0043], dtype=torch.float64), tensor([16,  3,  9,  1])]
batch: 4 [tensor([0.9598, 0.3659, 0.5615, 0.8089], dtype=torch.float64), tensor([12, 21, 13, 24])]
batch: 5 [tensor([0.4963, 0.9872, 0.8313, 0.7576], dtype=torch.float64), tensor([ 4, 27, 18, 26])]
batch: 6 [tensor([0.7196, 0.2364, 0.1819, 0.0236], dtype=torch.float64), tensor([14,  8, 20, 25])]
batch: 7 [tensor([0.8398, 0.7644, 0.1819, 0.6147], dtype=torch.float64), tensor([29,  0, 28, 23])]
Seeds at epoch 7: [24  3 56 72  0 21 19 74 41 10 21 38 96 20 44 93 39 14 26 81 90 22 66  2
 63 60  1 51 90 69]


k=2:
Pytorch: 1.0.0
Python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55)
[GCC 7.3.0]
num_workers 2
epoch 1
batch: 1 [tensor([0.2919, 0.5615, 0.0036, 0.6104], dtype=torch.float64), tensor([23, 26, 17, 27])]
batch: 2 [tensor([0.4736, 0.4978, 0.7380, 0.6558], dtype=torch.float64), tensor([ 7,  8,  4, 25])]
batch: 3 [tensor([0.0492, 0.7196, 0.0530, 0.6611], dtype=torch.float64), tensor([ 2,  0,  1, 18])]
batch: 4 [tensor([0.7518, 0.9873, 0.7731, 0.7731], dtype=torch.float64), tensor([22, 20, 12, 11])]
batch: 5 [tensor([0.4657, 0.3250, 0.0036, 0.4670], dtype=torch.float64), tensor([13, 24, 16, 14])]
batch: 6 [tensor([0.7911, 0.0036, 0.7731, 0.7380], dtype=torch.float64), tensor([21,  9, 19,  3])]
batch: 7 [tensor([0.8029, 0.1710, 0.9731, 0.4350], dtype=torch.float64), tensor([ 6, 29, 15, 10])]
Seeds at epoch 7: [44 47 64 67 67  9 83 21 36 87 70 88 88 12 58 65 39 87 46 88 81 37 25 77
 72  9 20 80 69 79]
epoch 2
batch: 1 [tensor([0.0036, 0.1625, 0.8303, 0.6180], dtype=torch.float64), tensor([24,  8,  5, 22])]
batch: 2 [tensor([0.5073, 0.5695, 0.0918, 0.7865], dtype=torch.float64), tensor([11, 21, 29, 17])]
batch: 3 [tensor([0.8674, 0.5615, 0.1628, 0.1710], dtype=torch.float64), tensor([28, 15, 26,  6])]
batch: 4 [tensor([0.0036, 0.9598, 0.1226, 0.3250], dtype=torch.float64), tensor([25, 27, 20,  2])]
batch: 5 [tensor([0.6180, 0.6275, 0.7911, 0.7518], dtype=torch.float64), tensor([14, 19,  0, 13])]
batch: 6 [tensor([0.5724, 0.4657, 0.5722, 0.7576], dtype=torch.float64), tensor([10,  1, 12,  9])]
batch: 7 [tensor([0.3991, 0.0492, 0.6558, 0.1490], dtype=torch.float64), tensor([23,  7,  3, 18])]
Seeds at epoch 7: [37 12 72  9 75  5 79 64 16  1 76 71  6 25 50 20 18 84 11 28 29 14 50 68
 87 87 94 96 86 13]
epoch 3
batch: 1 [tensor([0.6487, 0.7911, 0.7380, 0.2298], dtype=torch.float64), tensor([11, 18, 20, 15])]
batch: 2 [tensor([0.8718, 0.3250, 0.8089, 0.5596], dtype=torch.float64), tensor([ 8,  2, 14, 21])]
batch: 3 [tensor([0.4540, 0.0036, 0.7731, 0.0530], dtype=torch.float64), tensor([ 4, 19, 29, 13])]
batch: 4 [tensor([0.3659, 0.0469, 0.5737, 0.5615], dtype=torch.float64), tensor([ 3,  9, 10, 17])]
batch: 5 [tensor([0.4578, 0.8157, 0.8823, 0.5349], dtype=torch.float64), tensor([ 5, 12, 22,  7])]
batch: 6 [tensor([0.9872, 0.1819, 0.6186, 0.7380], dtype=torch.float64), tensor([23, 16, 25, 27])]
batch: 7 [tensor([0.8398, 0.6487, 0.2973, 0.5019], dtype=torch.float64), tensor([28,  6,  1, 24])]
Seeds at epoch 7: [40 15 72 22 43 82 75  7 34 49 95 75 85 47 63 31 90 20 37 39 67  4 42 51
 38 33 58 67 69 88]
epoch 4
batch: 1 [tensor([0.7638, 0.4736, 0.9531, 0.2267], dtype=torch.float64), tensor([ 7, 10,  2, 15])]
batch: 2 [tensor([0.4976, 0.4736, 0.9873, 0.5019], dtype=torch.float64), tensor([22,  5, 19, 11])]
batch: 3 [tensor([0.0036, 0.3250, 0.4581, 0.0043], dtype=torch.float64), tensor([16,  3,  9,  1])]
batch: 4 [tensor([0.9598, 0.3659, 0.5615, 0.8089], dtype=torch.float64), tensor([12, 21, 13, 24])]
batch: 5 [tensor([0.4963, 0.9872, 0.8313, 0.7576], dtype=torch.float64), tensor([ 4, 27, 18, 26])]
batch: 6 [tensor([0.7196, 0.2364, 0.1819, 0.0236], dtype=torch.float64), tensor([14,  8, 20, 25])]
batch: 7 [tensor([0.8398, 0.7644, 0.1819, 0.6147], dtype=torch.float64), tensor([29,  0, 28, 23])]
Seeds at epoch 7: [24  3 56 72  0 21 19 74 41 10 21 38 96 20 44 93 39 14 26 81 90 22 66  2
 63 60  1 51 90 69]


k=0:
Pytorch: 1.0.0
Python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55)
[GCC 7.3.0]
num_workers 0
epoch 1
batch: 1 [tensor([0.2919, 0.5615, 0.0036, 0.6104], dtype=torch.float64), tensor([23, 26, 17, 27])]
batch: 2 [tensor([0.4736, 0.4978, 0.7380, 0.6558], dtype=torch.float64), tensor([ 7,  8,  4, 25])]
batch: 3 [tensor([0.0492, 0.7196, 0.0530, 0.6611], dtype=torch.float64), tensor([ 2,  0,  1, 18])]
batch: 4 [tensor([0.7518, 0.9873, 0.7731, 0.7731], dtype=torch.float64), tensor([22, 20, 12, 11])]
batch: 5 [tensor([0.4657, 0.3250, 0.0036, 0.4670], dtype=torch.float64), tensor([13, 24, 16, 14])]
batch: 6 [tensor([0.7911, 0.0036, 0.7731, 0.7380], dtype=torch.float64), tensor([21,  9, 19,  3])]
batch: 7 [tensor([0.8029, 0.1710, 0.9731, 0.4350], dtype=torch.float64), tensor([ 6, 29, 15, 10])]
Seeds at epoch 7: [44 47 64 67 67  9 83 21 36 87 70 88 88 12 58 65 39 87 46 88 81 37 25 77
 72  9 20 80 69 79]
epoch 2
batch: 1 [tensor([0.0036, 0.1625, 0.8303, 0.6180], dtype=torch.float64), tensor([24,  8,  5, 22])]
batch: 2 [tensor([0.5073, 0.5695, 0.0918, 0.7865], dtype=torch.float64), tensor([11, 21, 29, 17])]
batch: 3 [tensor([0.8674, 0.5615, 0.1628, 0.1710], dtype=torch.float64), tensor([28, 15, 26,  6])]
batch: 4 [tensor([0.0036, 0.9598, 0.1226, 0.3250], dtype=torch.float64), tensor([25, 27, 20,  2])]
batch: 5 [tensor([0.6180, 0.6275, 0.7911, 0.7518], dtype=torch.float64), tensor([14, 19,  0, 13])]
batch: 6 [tensor([0.5724, 0.4657, 0.5722, 0.7576], dtype=torch.float64), tensor([10,  1, 12,  9])]
batch: 7 [tensor([0.3991, 0.0492, 0.6558, 0.1490], dtype=torch.float64), tensor([23,  7,  3, 18])]
Seeds at epoch 7: [37 12 72  9 75  5 79 64 16  1 76 71  6 25 50 20 18 84 11 28 29 14 50 68
 87 87 94 96 86 13]
epoch 3
batch: 1 [tensor([0.6487, 0.7911, 0.7380, 0.2298], dtype=torch.float64), tensor([11, 18, 20, 15])]
batch: 2 [tensor([0.8718, 0.3250, 0.8089, 0.5596], dtype=torch.float64), tensor([ 8,  2, 14, 21])]
batch: 3 [tensor([0.4540, 0.0036, 0.7731, 0.0530], dtype=torch.float64), tensor([ 4, 19, 29, 13])]
batch: 4 [tensor([0.3659, 0.0469, 0.5737, 0.5615], dtype=torch.float64), tensor([ 3,  9, 10, 17])]
batch: 5 [tensor([0.4578, 0.8157, 0.8823, 0.5349], dtype=torch.float64), tensor([ 5, 12, 22,  7])]
batch: 6 [tensor([0.9872, 0.1819, 0.6186, 0.7380], dtype=torch.float64), tensor([23, 16, 25, 27])]
batch: 7 [tensor([0.8398, 0.6487, 0.2973, 0.5019], dtype=torch.float64), tensor([28,  6,  1, 24])]
Seeds at epoch 7: [40 15 72 22 43 82 75  7 34 49 95 75 85 47 63 31 90 20 37 39 67  4 42 51
 38 33 58 67 69 88]
epoch 4
batch: 1 [tensor([0.7638, 0.4736, 0.9531, 0.2267], dtype=torch.float64), tensor([ 7, 10,  2, 15])]
batch: 2 [tensor([0.4976, 0.4736, 0.9873, 0.5019], dtype=torch.float64), tensor([22,  5, 19, 11])]
batch: 3 [tensor([0.0036, 0.3250, 0.4581, 0.0043], dtype=torch.float64), tensor([16,  3,  9,  1])]
batch: 4 [tensor([0.9598, 0.3659, 0.5615, 0.8089], dtype=torch.float64), tensor([12, 21, 13, 24])]
batch: 5 [tensor([0.4963, 0.9872, 0.8313, 0.7576], dtype=torch.float64), tensor([ 4, 27, 18, 26])]
batch: 6 [tensor([0.7196, 0.2364, 0.1819, 0.0236], dtype=torch.float64), tensor([14,  8, 20, 25])]
batch: 7 [tensor([0.8398, 0.7644, 0.1819, 0.6147], dtype=torch.float64), tensor([29,  0, 28, 23])]
Seeds at epoch 7: [24  3 56 72  0 21 19 74 41 10 21 38 96 20 44 93 39 14 26 81 90 22 66  2
 63 60  1 51 90 69]


k=1:
Pytorch: 1.0.0
Python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55)
[GCC 7.3.0]
num_workers 1
epoch 1
batch: 1 [tensor([0.2919, 0.5615, 0.0036, 0.6104], dtype=torch.float64), tensor([23, 26, 17, 27])]
batch: 2 [tensor([0.4736, 0.4978, 0.7380, 0.6558], dtype=torch.float64), tensor([ 7,  8,  4, 25])]
batch: 3 [tensor([0.0492, 0.7196, 0.0530, 0.6611], dtype=torch.float64), tensor([ 2,  0,  1, 18])]
batch: 4 [tensor([0.7518, 0.9873, 0.7731, 0.7731], dtype=torch.float64), tensor([22, 20, 12, 11])]
batch: 5 [tensor([0.4657, 0.3250, 0.0036, 0.4670], dtype=torch.float64), tensor([13, 24, 16, 14])]
batch: 6 [tensor([0.7911, 0.0036, 0.7731, 0.7380], dtype=torch.float64), tensor([21,  9, 19,  3])]
batch: 7 [tensor([0.8029, 0.1710, 0.9731, 0.4350], dtype=torch.float64), tensor([ 6, 29, 15, 10])]
Seeds at epoch 7: [44 47 64 67 67  9 83 21 36 87 70 88 88 12 58 65 39 87 46 88 81 37 25 77
 72  9 20 80 69 79]
epoch 2
batch: 1 [tensor([0.0036, 0.1625, 0.8303, 0.6180], dtype=torch.float64), tensor([24,  8,  5, 22])]
batch: 2 [tensor([0.5073, 0.5695, 0.0918, 0.7865], dtype=torch.float64), tensor([11, 21, 29, 17])]
batch: 3 [tensor([0.8674, 0.5615, 0.1628, 0.1710], dtype=torch.float64), tensor([28, 15, 26,  6])]
batch: 4 [tensor([0.0036, 0.9598, 0.1226, 0.3250], dtype=torch.float64), tensor([25, 27, 20,  2])]
batch: 5 [tensor([0.6180, 0.6275, 0.7911, 0.7518], dtype=torch.float64), tensor([14, 19,  0, 13])]
batch: 6 [tensor([0.5724, 0.4657, 0.5722, 0.7576], dtype=torch.float64), tensor([10,  1, 12,  9])]
batch: 7 [tensor([0.3991, 0.0492, 0.6558, 0.1490], dtype=torch.float64), tensor([23,  7,  3, 18])]
Seeds at epoch 7: [37 12 72  9 75  5 79 64 16  1 76 71  6 25 50 20 18 84 11 28 29 14 50 68
 87 87 94 96 86 13]
epoch 3
batch: 1 [tensor([0.6487, 0.7911, 0.7380, 0.2298], dtype=torch.float64), tensor([11, 18, 20, 15])]
batch: 2 [tensor([0.8718, 0.3250, 0.8089, 0.5596], dtype=torch.float64), tensor([ 8,  2, 14, 21])]
batch: 3 [tensor([0.4540, 0.0036, 0.7731, 0.0530], dtype=torch.float64), tensor([ 4, 19, 29, 13])]
batch: 4 [tensor([0.3659, 0.0469, 0.5737, 0.5615], dtype=torch.float64), tensor([ 3,  9, 10, 17])]
batch: 5 [tensor([0.4578, 0.8157, 0.8823, 0.5349], dtype=torch.float64), tensor([ 5, 12, 22,  7])]
batch: 6 [tensor([0.9872, 0.1819, 0.6186, 0.7380], dtype=torch.float64), tensor([23, 16, 25, 27])]
batch: 7 [tensor([0.8398, 0.6487, 0.2973, 0.5019], dtype=torch.float64), tensor([28,  6,  1, 24])]
Seeds at epoch 7: [40 15 72 22 43 82 75  7 34 49 95 75 85 47 63 31 90 20 37 39 67  4 42 51
 38 33 58 67 69 88]
epoch 4
batch: 1 [tensor([0.7638, 0.4736, 0.9531, 0.2267], dtype=torch.float64), tensor([ 7, 10,  2, 15])]
batch: 2 [tensor([0.4976, 0.4736, 0.9873, 0.5019], dtype=torch.float64), tensor([22,  5, 19, 11])]
batch: 3 [tensor([0.0036, 0.3250, 0.4581, 0.0043], dtype=torch.float64), tensor([16,  3,  9,  1])]
batch: 4 [tensor([0.9598, 0.3659, 0.5615, 0.8089], dtype=torch.float64), tensor([12, 21, 13, 24])]
batch: 5 [tensor([0.4963, 0.9872, 0.8313, 0.7576], dtype=torch.float64), tensor([ 4, 27, 18, 26])]
batch: 6 [tensor([0.7196, 0.2364, 0.1819, 0.0236], dtype=torch.float64), tensor([14,  8, 20, 25])]
batch: 7 [tensor([0.8398, 0.7644, 0.1819, 0.6147], dtype=torch.float64), tensor([29,  0, 28, 23])]
Seeds at epoch 7: [24  3 56 72  0 21 19 74 41 10 21 38 96 20 44 93 39 14 26 81 90 22 66  2
 63 60  1 51 90 69]


k=3:
Pytorch: 1.0.0
Python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55)
[GCC 7.3.0]
num_workers 3
epoch 1
batch: 1 [tensor([0.2919, 0.5615, 0.0036, 0.6104], dtype=torch.float64), tensor([23, 26, 17, 27])]
batch: 2 [tensor([0.4736, 0.4978, 0.7380, 0.6558], dtype=torch.float64), tensor([ 7,  8,  4, 25])]
batch: 3 [tensor([0.0492, 0.7196, 0.0530, 0.6611], dtype=torch.float64), tensor([ 2,  0,  1, 18])]
batch: 4 [tensor([0.7518, 0.9873, 0.7731, 0.7731], dtype=torch.float64), tensor([22, 20, 12, 11])]
batch: 5 [tensor([0.4657, 0.3250, 0.0036, 0.4670], dtype=torch.float64), tensor([13, 24, 16, 14])]
batch: 6 [tensor([0.7911, 0.0036, 0.7731, 0.7380], dtype=torch.float64), tensor([21,  9, 19,  3])]
batch: 7 [tensor([0.8029, 0.1710, 0.9731, 0.4350], dtype=torch.float64), tensor([ 6, 29, 15, 10])]
Seeds at epoch 7: [44 47 64 67 67  9 83 21 36 87 70 88 88 12 58 65 39 87 46 88 81 37 25 77
 72  9 20 80 69 79]
epoch 2
batch: 1 [tensor([0.0036, 0.1625, 0.8303, 0.6180], dtype=torch.float64), tensor([24,  8,  5, 22])]
batch: 2 [tensor([0.5073, 0.5695, 0.0918, 0.7865], dtype=torch.float64), tensor([11, 21, 29, 17])]
batch: 3 [tensor([0.8674, 0.5615, 0.1628, 0.1710], dtype=torch.float64), tensor([28, 15, 26,  6])]
batch: 4 [tensor([0.0036, 0.9598, 0.1226, 0.3250], dtype=torch.float64), tensor([25, 27, 20,  2])]
batch: 5 [tensor([0.6180, 0.6275, 0.7911, 0.7518], dtype=torch.float64), tensor([14, 19,  0, 13])]
batch: 6 [tensor([0.5724, 0.4657, 0.5722, 0.7576], dtype=torch.float64), tensor([10,  1, 12,  9])]
batch: 7 [tensor([0.3991, 0.0492, 0.6558, 0.1490], dtype=torch.float64), tensor([23,  7,  3, 18])]
Seeds at epoch 7: [37 12 72  9 75  5 79 64 16  1 76 71  6 25 50 20 18 84 11 28 29 14 50 68
 87 87 94 96 86 13]
epoch 3
batch: 1 [tensor([0.6487, 0.7911, 0.7380, 0.2298], dtype=torch.float64), tensor([11, 18, 20, 15])]
batch: 2 [tensor([0.8718, 0.3250, 0.8089, 0.5596], dtype=torch.float64), tensor([ 8,  2, 14, 21])]
batch: 3 [tensor([0.4540, 0.0036, 0.7731, 0.0530], dtype=torch.float64), tensor([ 4, 19, 29, 13])]
batch: 4 [tensor([0.3659, 0.0469, 0.5737, 0.5615], dtype=torch.float64), tensor([ 3,  9, 10, 17])]
batch: 5 [tensor([0.4578, 0.8157, 0.8823, 0.5349], dtype=torch.float64), tensor([ 5, 12, 22,  7])]
batch: 6 [tensor([0.9872, 0.1819, 0.6186, 0.7380], dtype=torch.float64), tensor([23, 16, 25, 27])]
batch: 7 [tensor([0.8398, 0.6487, 0.2973, 0.5019], dtype=torch.float64), tensor([28,  6,  1, 24])]
Seeds at epoch 7: [40 15 72 22 43 82 75  7 34 49 95 75 85 47 63 31 90 20 37 39 67  4 42 51
 38 33 58 67 69 88]
epoch 4
batch: 1 [tensor([0.7638, 0.4736, 0.9531, 0.2267], dtype=torch.float64), tensor([ 7, 10,  2, 15])]
batch: 2 [tensor([0.4976, 0.4736, 0.9873, 0.5019], dtype=torch.float64), tensor([22,  5, 19, 11])]
batch: 3 [tensor([0.0036, 0.3250, 0.4581, 0.0043], dtype=torch.float64), tensor([16,  3,  9,  1])]
batch: 4 [tensor([0.9598, 0.3659, 0.5615, 0.8089], dtype=torch.float64), tensor([12, 21, 13, 24])]
batch: 5 [tensor([0.4963, 0.9872, 0.8313, 0.7576], dtype=torch.float64), tensor([ 4, 27, 18, 26])]
batch: 6 [tensor([0.7196, 0.2364, 0.1819, 0.0236], dtype=torch.float64), tensor([14,  8, 20, 25])]
batch: 7 [tensor([0.8398, 0.7644, 0.1819, 0.6147], dtype=torch.float64), tensor([29,  0, 28, 23])]
Seeds at epoch 7: [24  3 56 72  0 21 19 74 41 10 21 38 96 20 44 93 39 14 26 81 90 22 66  2
 63 60  1 51 90 69]

I tested a on real code X (Pytorch 1.0.0) where the worker need to do some random operations on each sample such as cropping, rotation, and so on.
Repeating the same code X twice (same device: GPU id) using the same number of workers leads to the same exact results (100% reproducible). However, changing the number of workers leads to entirely different results (normal behavior). Using the workaround mentioned above: the code X remains reproducible when repeated using the same number of workers, and produces exactly the same results whatever the number of workers k>=0. HOWEVER, the example discussed in this post is very basic and shows only the reproducibility over the dataloader. In practice, the main process does many other things that involves randomness (which we didn’t show here). Such randomness will cause a problem when k=0 compared to k>0.
Now, we understand that the job of a worker is only to load samples. The reste of the code is done by the main process. The thing that we didn’t show in the above example, and which is the case in practice, is the following:
# P0
for i, (data, labels) in enumerate(dataloader):  # calling workers.
  # P1
  # do some random stuff: main process.

In this case, the state of the internal generator of the main process at P1 will depend on k:

If k>0: the state at P0 is the same at P1.
If k=0: the state at P0 is different than the state at P1 because it is the main process who loads the samples. Therefore, if that loading needs randomness, the state of the main process will be different after loading the samples.

To avoid such issue, you need to make sure that after calling the code of the workers, you need to reset the state of the main process to a fixed state (it does not matter if it changes from iteration to another). The most important is that the state of the main process at P1 is the same independently of k. Here is a snippet:
# P0
for i, (data, labels) in enumerate(dataloader):  # calling workers.
  # P1: re-seed the state of the main process to something that does not depend on the number k.
  # do some random stuff: main process.

You need to do this re-seeding of the state of the main process at every access to the dataloader (every call to the workers). In practice, this is mostly done when looping over minibatches (train/evaluation).