Thanks for the reply, @ptrblck , I’ve learned a lot from reading it.
However, after profiling my code (with model training), I’ve still witnessed the same thing.
I used tensorboard together with pytorch lightning’s profiler to measure the inter step time (the time between one step’s end, and another’s start).
The weird behavior persists. It starts a bit after step 20, and then you see periodical slows in the fetching of data, with num_workers=8 being the period time. 
My Full dataset code is:

class NumpyDataset(torch.utils.data.Dataset):
    """A pytorch dataset based on a folder of npy files"""
    def __init__(self, root, families, chroms, indices,
                 transform=None):
        self.root = Path(root)

        self.npy_files_args = list(itertools.product(families, chroms, indices))
        self.num_files = len(self.npy_files_args)
        # we define the transform attribute
        self.transform = transform

        # # we set a pseudo-random permutation
        np.random.seed(0)
        self.random_permutation = np.random.permutation(self.num_files)

    def get_file_path(self, index):
        fam, chrom, idx = self.npy_files_args[index]
        return self.root / fam / chrom / f'{fam}_{chrom}_{idx:05d}.npy'

    def __getitem__(self, index):
        # we use the given index to extract the sample at the permuted index
        permuted_idx = self.random_permutation[index]

        sample = np.load(self.get_file_path(permuted_idx), allow_pickle=False)
        # except:
        #     print(f'Failed to load file {self.get_file_path(permuted_idx)}')
        #     return None

        # if transform is given, we apply it
        if self.transform:
            sample = self.transform(sample)

        return sample

    def __len__(self):
        return self.num_files

If anyone knows why this behavior persists, I would love to know.
P.S The data consists of a 3d tensor + 2d tensor + little numpy arrays with some metadata information. Might the transformations on this data be heavy for the dataloaders?