Thanks, the other only alternative that I came up with is to use the transforms as before without the normalization
transforms.Resize(32),
transforms.RandomHorizontalFlip(),                        
transforms.RandomCrop(32, padding=4),                     
transforms.ToTensor(),                                     
),

then when iterating over the loader checking channel dim and repeating accordingly
for x, y in loader:
  if x.shape[1] != 3:
     x = x.repeat(1, 3, 1, 1)