Background:

This is on an nvidia DGX2. I confirmed that all data is cached on the NVMe cache (as the cache does not grow as training progresses)
I removed DataParallel()/DDP()/AMP, and am training on one GPU as @mcarilli suggested
All hyperparams including minibatch and n_workers for the dataloader are kept the same for all tests
I invoke time.time() after each iteration to get the seconds per iteration.

Profiling results:


While training with a small dataset (4k samples), it takes 1.2 seconds per iteration, and that speed is consistent after tens of thousands of iterations


While training with a large dataset (65k samples), it takes an average of 2 seconds per iteration. The way the slowness manifests is a handful of “fast” iteration at 1.2 seconds/iter, followed by a slow iteration that takes 4-10 seconds. It averages out close to 2 seconds/iter. When scaling up to many GPUs, the speed penalty (average seconds/iteration) with the large dataset is 5x or more (and only a negligible speed penalty with a small dataset).


To profile the RAM, I am calling the watch free command. It stays fairly consistent (doesn’t blow up as training progresses):


              total        used        free      shared  buff/cache   available
Mem:     1583405972    32676452     1541340      351212  1549188180  1548865308
Swap:       3904508      279808     3624700



I imagine the slowness is from loading the data, but I will try the elapsed_time() profiler as well as @mrshenli suggested


@jspisak I have not yet filed a GitHub issue - let me know if I should do that