Hi
I’m currently running a small test network, which consist of 378 parameters.
The network is tested on a dataset which consist of 600 points, with 2 features each (points in 2D).
All the data is loaded into the standard pytorch dataloader, and I keep it all on cpu and does not employ nn.dataparallel on my dataloader in this model.
Now the problem comes when I iterate over the dataloader, for some reason this takes more than 99% of the total computational time, I do not apply any data transformation or use any complicated datasampler.
This is what my dataloader looks like:
    train_loader = data.DataLoader(dataset_train_torch, batch_size=args.batch_size, shuffle=True,
                            num_workers=20, pin_memory=True, drop_last=True)

For one training epoch I get timings that looks like this: time_data: 0.0007 	time_tot: 6.4737
Where time_tot is the full time for an epoch, and time_data is the time for doing everything after the dataloading. My batch size is 100.
Is the dataloader just not well suited for such small datasets?
Even then how can almost 6.5 s be used just loading the data at each epoch?
Any help would be greatly appreciated.