Could you try to load some example images directly via PIL.Image.open and see, if numpy arrays would have the expected shape?
If so, you could provide a custom loader function instead of the pil_loader and just remove the convert('RGB').