Fixed num_channels!=3 normalization training [#20630]
# What does this PR do?

Fork from #20630 (which is to fix https://github.com/huggingface/transformers/issues/20580 and https://github.com/huggingface/transformers/issues/19913)
_The documentation is not available anymore as the PR was closed or merged._<|||||>@NielsRogge CI is green, but I don't review (yet) the content of PR though<|||||>@NielsRogge 

I think it is already there: when I clicked `Squash and merge`, it shows at the end

```
Co-authored-by: Lay Jain <layjain@basil.csail.mit.edu>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
```<|||||>cc @layjain <|||||>The CI got fixed on the other PR so I merged it. Is there still a need for this one?<|||||>@sgugger OK, this morning it was not running even I pushed a commit. No more need of this PR - despite I have a question regarding the logic. We can fix it if @NielsRogge think what I pointed is indeed a bug.