I guess all three channels have the same values then. You could check if that’s the case and just keep a single channel, since the other two won’t give you any more information, if you would like to train from scratch.
Regarding your loss curves, I would try to overfit a small subset of your dataset (e.g. 1-10 samples) and see if that’s working. If your model can’t fit this small sample, you might have some other bugs in your code. However, if it’s working, you could try to carefully scale up your experiment and play with some hyperparameters.