Hello, I have faced a problem with num_workers while training my models. I have made a simple transfer learning task with densenet121 and CIFAR10 resized to the ImageNet resolution of 224:
class DenseNet(nn.Module):
    def __init__(self):
        super(DenseNet, self).__init__()
        
        # densenet
        self.densenet = densenet121(pretrained=False)
        
        # classifier
        self.densenet.classifier = nn.Linear(in_features=self.densenet.classifier.in_features, out_features=10)
        
    def forward(self, x):
        
        return self.densenet(x)

Transform and the data loader are defined as follows:
# transform
transform = transforms.Compose([transforms.Resize((224, 224)), 
                                transforms.ToTensor()])

# define the training data loader
training_loader = DataLoader(dataset=cifar_dataset, batch_size=64, shuffle=True, pin_memory=True, num_workers=8)

So, I use 8 workers on the host to feed the data to the device. The model works just fine, however, I have noticed that not all workers do something (in fact, all 8 don’t seem to do anything), while the main thread is the one that loads the data (loads up to 10% though):

dead_workers.PNG799×243 5.39 KB

So, it looks like this creates a bottleneck and the GPU isn’t utilized for more than 15%:

If anyone knows what is going on, please advise.
Specs:

CPU: AMD 2700X
GPU: GTX 1080Ti
The dataset is on SSD
CUDA: 10.0
cuDNN: 7.42