Thanks a lot. I was hoping to avoid changing the model by adding an additional conv operation just to make the input sizes appropriate.
I tried having it in my transforms
transforms.Resize(32),
transforms.RandomHorizontalFlip(),                        
transforms.RandomCrop(32, padding=4),                     
transforms.ToTensor(),                                    
transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
transforms.Lambda(lambda x: nn.Conv2d(                    
in_channels=1, out_channels=3, kernel_size=1)(x)      
),

but produces broadcast errors
Traceback (most recent call last):
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torchvision/datasets/mnist.py", line 95, in __getitem__
    img = self.transform(img)
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 166, in __call__
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 217, in normalize
    tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
RuntimeError: output with shape [1, 32, 32] doesn't match the broadcast shape [3, 32, 32]

I seems that what I’m missing is the equivalent commands of
train_set.data.unsqueeze_(1)                                                                                                                                                                 
train_set.data = train_set.data.repeat(1, 3, 1, 1)  

but for PIL images so that I can hook them up in the transforms.