I verified this is not a data-pinning issue. Ok to close this thread based on the title.
If the GPU is not used in the DataLoader test, the transfer rate for a single worker is still the same.

def DataLoader_Test(torch_loader, num_workers):
torch.cuda.synchronize()
t_start = time.time()
for x in torch_loader:
sample = x
#sample = x.to(‘cuda’, non_blocking=True)
torch.cuda.synchronize()
t_end = time.time()
t_time = (t_end - t_start)*1000
t_size = n_samples*(Batch*N*C*W)*4/1024/1024
t_bw = t_size/(t_end-t_start)

print('DataLoader Test: ', num_workers, ' workers')
print('size of transfer ', t_size, 'MB')
print('time taken by transfer ', t_time, 'mSec')
print('Effective bandwidth ', t_bw, 'MBps (DataLoader:',num_workers,' workers)')


Result:

DataLoader Test:  0  workers
size of transfer  244140.625 MB
time taken by transfer  8440.999269485474 mSec
Effective bandwidth  28923.18992166928 MBps (DataLoader: 0  workers)
DataLoader Test:  1  workers
size of transfer  244140.625 MB
time taken by transfer  127765.62309265137 mSec
Effective bandwidth  1910.847527608873 MBps (DataLoader: 1  workers)

I did look at the dataloader construction and it is loading a queue and then passing through a second queue if being pinned. It must be the queue throughput and pinning overhead that is limiting performance.
If the pin_memory is set to False, and again bypassing the GPU:

num_workers=0
torch_loader = DataLoader(pytorch_dataset(w1),
batch_size=None,
num_workers=num_workers,
pin_memory=False)
DataLoader_Test(torch_loader, num_workers)
num_workers=1
torch_loader = DataLoader(pytorch_dataset(w1),
batch_size=None,
num_workers=num_workers,
pin_memory=False)
DataLoader_Test(torch_loader, num_workers)

The results show bandwidth well above, where the single worker instance is passing through a single queue, but not the second pinning queue:

DataLoader Test:  0  workers
size of transfer  244140.625 MB
time taken by transfer  5.549430847167969 mSec
Effective bandwidth  43993813.36999484 MBps (DataLoader: 0  workers)
DataLoader Test:  1  workers
size of transfer  244140.625 MB
time taken by transfer  14370.611906051636 mSec
Effective bandwidth  16988.881656263326 MBps (DataLoader: 1  workers)