ImageFolder will use the default_loader, which would then either use accimage_loader, if it’s available, or pil_loader, which will convert the loaded images to RGB as seen here 31.
You can create a custom loader by e.g. just reusing the pil_loader and removing the convert call, and pass it into the loader argument in ImageFolder.