Fixed num_channels!=3 normalization training
Fixes #20580 and #19913 

## Who can review?

@NielsRogge 

_The documentation is not available anymore as the PR was closed or merged._<|||||>Not exactly, the issue with your CircleCI permissions, the tests won't run.

Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?<|||||>I have been now stuck on this for a while. I refreshed the permissions and re-ran the CircleCI checks, and I get the error:
"Resource class docker for xlarge is not available for your project, or is not a valid resource class. This message will often appear if the pricing plan for this project does not support docker use."


<|||||>You might need to push an empty commit to re-trigger the tests after refreshing your permissions.<|||||>Hi @layjain let me know whether you could pick this up :)<|||||>FYI I pushed an empty commit to trigger CI<|||||>Hi @NielsRogge , I have fixed the CircleCI permissions, can this be merged.<|||||>@layjain The CI is currently running under your profile and not the Hugging Face profile, and as such our tests are mostly not run (there should be 22 checks here). If you rebase your PR on main you will see a new check failing (we added a fix to detect this recently).