Hi folks,
I’ve noticed a couple of things going on with DataLoader.

Transofrms are only invoked when train_loader iterator is invoked in the training loop. This is quite expensive. I dunno if this is intended behavriour?

# Image Preprocessing                                                                                                                                                       
transform = trans.Compose([                                                                                                                                                 
    trans.ToTensor(),                                                                                                                                                       
    trans.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),                                                                                                             
])                                                                                                                                                                          
                                                                                                                                                                            
# MNIST Dataset                                                                                                                                                             
train_dataset = dsets.MNIST(root='./data/mnist/',                                                                                                                           
                            train=True,                                                                                                                                     
                            transform=transform,                                                                                                                            
                            download=False)                                                                                                                                 
                                                                                                                                                                            
# Data Loader (Input Pipeline)                                                                                                                                              
train_loader = torch.utils.data.DataLoader(                                                                                                                                 
    dataset=train_dataset,                                                                                                                                                  
    batch_size=train_dataset.train_data.shape[0],                                                                                                                           
    shuffle=True,                                                                                                                                                           
    num_workers=10,                                                                                                                                                         
    pin_memory=True                                                                                                                                                         
)

import pdb; pdb.set_trace()
# Now examine the train_dataset.train_data                     



The train_dataset.train_data is scaled between [-1, +1] shouldn’t that be between [0, 1]. Since the scaling happens automatically via ToTensor() invocation how can we tell ToTensor() to scale between [0, 1]?


This might not be correct but here’s the training example trying to use data loader to perform full batch:


for epoch in range(200):                                                                                                                                                    
    for i, (images, labels) in enumerate(train_loader):                                                                                                                     
        # Build mini-batch dataset                                                                                                                                          
        images = images.cuda(1)                                                                                                                                             
        labels = labels.cuda(1)                                                                                                                                             
        images = images.view(images.size(0), -1) 

Using this approach of data loader + full batch training I get the following speeds:


If I bypass completely the data loader and do the full batch training on my own here’s what I get:



Am I doing sth fundamentally wrong here?
Maybe It would be nice for the data loader to have flag indicating to load all data in memory if possible?