Since you already have your data in numpy, I would suggest to build an own Dataset, instead of using ImageFolder.
Here 1.9k is a good example how to create a Dataset.
Also your image data seems to be flat. Try to reshape it to [batch, channel, width, height].