Permuting the output is the right approach.
What do you mean by “not performing well”? Is the model bad regarding the speed or accuracy?
You could use the channels_last memory format as described here 398, which internally permutes the data to the channels last format. Note that the shape of the tensor would still indicate the standard contiguous format (channels first).