@jekbradbury
Thank you so much for your replies, I appreciate it. It cleared up a lot of stuff. I still have some questions maybe other people can pop in to the conversation.
Validation set:
The following code is how we load the CIFAR10 dataset. For test, we just set train=False

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
train_set = datasets.CIFAR10(root=expanduser(
    '~/learning/cifar10-data'), train=True, download=True, transform=transform)
train_loader = DataLoader(train_set, batch_size=4,
                          shuffle=True, num_workers=2)


My initial intuition was just to set train_loader = train_loader[:small_number] but I got an error:

train_loader[:100]
Traceback (most recent call last):
File “”, line 1, in 
TypeError: ‘DataLoader’ object is not subscriptable

Then I thought I could mess with the train_set directly but I got another error:

train_set[:100]
Traceback (most recent call last):
File “”, line 1, in 
File “/home/sudarshan/anaconda3/envs/torch/lib/python3.6/site-packages/torchvision-0.1.7-py3.6.egg/torchvision/datasets/cifar.py”, line 89, in getitem
File “/home/sudarshan/anaconda3/envs/torch/lib/python3.6/site-packages/numpy/core/fromnumeric.py”, line 550, in transpose
return _wrapfunc(a, ‘transpose’, axes)
File “/home/sudarshan/anaconda3/envs/torch/lib/python3.6/site-packages/numpy/core/fromnumeric.py”, line 57, in _wrapfunc
return getattr(obj, method)(*args, **kwds)
ValueError: axes don’t match array

Both these objects have a len function:

len(train_set)
50000
len(train_loader)
12500

So I’m not sure how to get a validation set out of this.
Train/Test mode:
According to the docs 3, eval has effect only on dropout and batch norm, which makes sense since their functions differ during testing as opposed to training. Further, we don’t explicitly set model.train() or model.eval(), when the testing is happening in the prediction code block.
So where is this flag being set and how do we know its not training again? I can think of two reasons on this works, but I’m not sure which one:


While loading the test_set, the train flag is set to False. Since testing is done on the test_loader (which was instantiated using the test_set), the mode was already set to “test” and testing automatically happened.


We just didn’t calculate loss, the gradients, and update the gradients through the optimization step. Take a look at the code blocks for training and testing:


Training block:

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()


Test block:

    outputs = net(images)
    _, predicted = torch.max(outputs.data, 1)
    total += labels.size(0)
    correct += (predicted == labels.data).sum()


During testing (aka prediction) we don’t compute the loss, run the backward, and run the optimization.step() which would mean we are just getting the class labels. So by omitting those steps we do the prediction? This makes sense to me after thinking about it, but it would be helpful if I could get confirmation that this is in fact what is happening.
So end of the day question is lets say we have loaded our dataset using standard numpy techniques and converted them into torch Tensors and have (X_train, y_train, X_test, y_test). How we specify when using X_test, y_test do testing as opposed to training (which would just be calculate loss, its gradients and update weights).
Thanks.