If X contains [N, H, W, C] data and you would like to provide it as [N, C, H, W], then your permutation is correct.
The constant output might come from a bad training run (overfitting to the majority class or mean value in a regression?).