In most cases, alpha channel is usually not needed for the model to work. So RGB is probably sufficient in your case as well. To be sure try to plot the image once you get from your the dataloader and verify that they have not lost lot of information (for the model to discriminate).