@mcarilli Thanks for the reply! On one GPU, no DP/DDP/Amp with 6 workers I observe the slowness with the large (65k sample) dataset, but increasing the number of workers to 12 removes the slowness (e.g. training with the large dataset is just as fast as training with the smaller dataset). Wrapping the networks with DataParallel() but keeping it at 1GPU/12 workers/minibatch=12 did not cause any slowness. When I scale up to 2 GPUs, num_workers=24 and using minibatch=24, I start to see the slowness again. The pattern of slow training is as described above - several “fast” iterations followed by a slow iteration of 5-50 seconds, and this behavior persists for tens of thousands of iters.
I tried using AMP on 1 GPU, effectively increasing the minibatch size from 12 to 30, and I observed the slow training with the large dataset even after increasing the number of workers to 24. With a small dataset of ~4K samples, only 3 workers are required and I get consistently fast speeds.
The machine has 96 CPUs and 16 GPUs so I would think 6 workers per GPU would be ideal. When working with the small dataset, only 3 workers are required to get peak training speeds.
The dataset is 2.8TB total spanning approx. 40 LMDBs
I have set torch.set_num_threads(96) and torch.set_num_interop_threads(48) at the beginning of the training script.
Whether I set pin_memory=True or False doesn’t change the behavior. I’m passing in the SequentialSampler() into the dataloader, but the slow training still happens if I set sampler=False and shuffle=True.
Here’s how I open each LMDB in the dataset’s __init__ class:
self.env = lmdb.open(lmdbPath, readonly=True, lock=False, readahead=False, meminit=False, max_readers=512)


readahead=True made things worse, meminit doesn’t appear to be a factor, and increasing max_readers didn’t help either
Here’s how I am reading imgs from the LMDB in the dataset’s __getitem__ method:

    with env.begin(write=False) as txn:
        buf = txn.get(path.encode('ascii'))
        buf_meta = txn.get((path + '.meta').encode('ascii')).decode('ascii')

Please do let me know of any suggestions! I think the key finding here is that only 3 workers are required for top training speeds with a small dataset and 1 GPU, but 12 were required with the large dataset (with same minibatch/hyperparams). With effectively larger minibatches from multi-GPU or AMP, I still saw slowness after increasing the num_workers beyond 12.