Thanks for you help!
I want to try to implement a custom pil loader but I am currently struggling a bit with this.
My plan was to add the custom pil loader within folder.py.
So I copied the normal pil_loader function, where ptrblck showed me it is located, and pasted it right above it.
I renamed it to custom_pil_loader and removed the convert call.
def custom_pil_loader(path):
# open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img

def pil_loader(path):
# open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')

After this I thought that it would be enough to call the new loader during the ImageFolder call.
I tried this here but it didn`t worked.
Could you tell me what I am doing wrong?
 train_ds = ImageFolder(os.path.join(opt.dataroot, 'train'), transform, loader='custom_pil_loader')
 valid_ds = ImageFolder(os.path.join(opt.dataroot, 'test'), transform, loader='custom_pil_loader')

That was the way I called the ImageFolder before what worked well, with the problem of the conversion in RGB of single channel images.
train_ds = ImageFolder(os.path.join(opt.dataroot, 'train'), transform)
valid_ds = ImageFolder(os.path.join(opt.dataroot, 'test'), transform)

I tried then to access the first batch of images within the dataloader
train_dl = DataLoader(dataset=train_ds, batch_size=opt.batchsize, shuffle=True, drop_last=True)
valid_dl = DataLoader(dataset=valid_ds, batch_size=opt.batchsize, shuffle=False, drop_last=False)

I run the following command
iter(train_dl).next()[0].shape

With the default_loader I get the following output
Out[1]: torch.Size([32, 3, 32, 32])

But when I try to run this command using my custom_pil_loader I get the following error message
Traceback (most recent call last):
  File "/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 
3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-405944bb4251>", line 1, in <module>
    iter(train_dl).next()[0].shape
  File "/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 
346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, 
in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, 
in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 
138, in __getitem__
    sample = self.loader(path)
TypeError: 'str' object is not callable