Yeah, so from the top of my head, I think the books mentions that depending on your problem type, you’d take (per-channel) averages over the entire Dataset. This is what is usually done for image classifiers (ImageNet).
I’m sure you can get good results without splitting by channel, given that the mean is something like [0.485, 0.456, 0.406] and the std [0.229, 0.224, 0.225], so they’re not that far apart.
When you use the batch, it is a bit funny because the image depends on its neighbors, I’d probably avoid that just because I’d feel dirty if I did that. On the other hand batch norm is arguably quite successful.
Best regards
Thomas