Disable the shuffling in the DataLoader (if not already done) and print the batch index to further isolate the problematic sample (also keep batch_size=1).
With this index you can directly index the Dataset and check why this sample has 3 channels only.