If I add an artificial bottleneck in the data loading part using this Dataset:
class MyDataset(Dataset):
    def __init__(self, size):
        self.size = size
        
    def __getitem__(self, index):
        x = torch.randn(3, 224, 224)
        y = torch.randint(0, 1000, (1,)).squeeze()
        
        # Simulate slow data loading
        a = torch.randn(100)
        for _ in range(1000):
            for _ in range(100):
                a = a * a
        return x, y
    
    def __len__(self):
        return self.size

I get an utilization of approx. 13%.
It might be a good idea to time your data loading using the ImageNet example 185.