ptrblck:

Did you apply the same transformations in the second case? Also, did you shuffle the data somehow?


That was missing in the first comparison.
Now with same transformations and very naive shuffling on full batch:

And here’s with the DataLoader:




 ptrblck:

Could you share your code to compare both approaches?


Sure thing, the code below runs full-batch, if you un-comment the commented lines and then it should run using DataLoader. When running with DataLoader comment the lines above the for loop for images, labels etc.:
import torch                                                                                                                                                                
import torch.nn as nn                                                                                                                                                       
import torch.nn.functional as F                                                                                                                                             
import torchvision.datasets as dsets                                                                                                                                        
import torchvision.transforms as trans                                                                                                                                                                                                                                                                          
                                                                                                                                                                            
# Image Preprocessing                                                                                                                                                       
transform = trans.Compose([                                                                                                                                                 
    trans.ToTensor(),                                                                                                                                                                                                                                                             
])                                                                                                                                                                          
                                                                                                                                                                            
# MNIST Dataset                                                                                                                                                             
train_dataset = dsets.MNIST(root='./data/mnist/',                                                                                                                           
                            train=True,                                                                                                                                     
                            transform=transform,                                                                                                                            
                            download=False)                                                                                                                                 
                                                                                                                                                                            
# Data Loader (Input Pipeline)                                                                                                                                              
train_loader = torch.utils.data.DataLoader(                                                                                                                                 
    dataset=train_dataset,                                                                                                                                                  
    batch_size=train_dataset.train_data.shape[0],                                                                                                                           
    shuffle=True,                                                                                                                                                           
    num_workers=10,                                                                                                                                                         
    pin_memory=True                                                                                                                                                         
)                                                                                                                                                                           
                                                                                                                                                                            
# MLP                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
class MLP(nn.Module):                                                                                                                                                                                                                                                                                                                          
    def __init__(self):                                                                                                                                                     
        super(MLP, self).__init__()                                                                                                                                         
        self.fc1 = nn.Linear(784, 512)                                                                                                                                      
        self.fc2 = nn.Linear(512, 256)                                                                                                                                      
        self.fc3 = nn.Linear(256, 128)                                                                                                                                      
        self.fc4 = nn.Linear(128, 10)                                                                                                                                       
                                                                                                                                                                            
    def forward(self, x):                                                                                                                                                   
        h = F.relu(self.fc1(x))                                                                                                                                             
        h = F.relu(self.fc2(h))                                                                                                                                             
        h = F.relu(self.fc3(h))
        out = F.log_softmax(self.fc4(h), dim=-1)                                                                                              
        return out                                                                                                                                                          
                                                                                                                                                                            
                                                                                                                                                                          
model = MLP()                                                                                                                                                               
# criterion = nn.CrossEntropyLoss()                                                                                                                                         
criterion = nn.NLLLoss()                                                                                                                                                    
optim = torch.optim.Adam(model.parameters(), lr=5e-4)                                                                                                                       
model.cuda(1)                                                                                                                                                               
                                                                                                                                                                            
                                                                                                                                                                            
train_dataset.train_data = train_dataset.train_data.float().div(255.0)                                                                                                    
images = train_dataset.train_data.cuda(1)                                                                                                                                 
labels = train_dataset.train_labels.cuda(1)                                                                                                                               
                                                                                                                                                                            
# Training                                                                                                                                                                  
for epoch in range(200):                                                                                                                                                    
    # for i, (images, labels) in enumerate(train_loader):                                                                                                                     
        # Build mini-batch dataset                                                                                                                                          
        # images = images.cuda(1)                                                                                                                                             
        # labels = labels.cuda(1)                                                                                                                                             
        images = images.view(images.size(0), -1)                                                                                                                            
        temp = torch.cat([                                                                                                                                                
            images,                                                                                                                                                       
            labels.view(images.size(0), 1).to(torch.float)                                                                                                                
        ], dim=1)                                                                                                                                                         
        idx = torch.randperm(temp.nelement())                                                                                                                             
        temp.view(-1)[idx].view(temp.size())                                                                                                                              
        images = temp[:, :images.size(1)]                                                                                                                                 
        labels = temp[:, -1].to(torch.long)                                                                                                                               
                                                                                                                                                                            
        # Train the model                                                                                                                
        outputs = model(images)                                                                                                                                             
        loss = criterion(outputs, labels)                                                                                                                                   
                                                                                                                                                                            
        loss.backward()                                                                                                                                                     
        optim.step()                                                                                                                                                        
        optim.zero_grad()
                                                                                                                                               
        print('Epoch [%d/%d], loss: %.4f,'                                                                                                                                  
              % (epoch, 200, loss.data))