I am trying to train a ViT model modification on the ImageNet dataset from scratch.
I am using 8 Teslas V100 GPUs and it is taking enormously too long.
While inspecting the gpus with nvidia-smi I get:

WhatsApp Image 2021-05-26 at 9.59.37 AM719×579 124 KB

I am using nn.DataParallel to train it. In my dataloader I am using num_workers = 8 and pin_memory=True of course.
I tried to increase the number of workers up to 16 as adviced in Guidelines for assigning num_workers to DataLoader - #4 by YossiB 2 but it only froze the machine.
Is that normal? The estimated time per epoch is around 9 hours, I think that’s too long, specially because I intend to train it for 300 epochs