By default the pil_loader would be used, which returns images in RGB and thus adds the channel.
You could provide another loader to ImageFolder and make sure the images are loaded in the desired format.