Thanks for the clarification. I’m not sure if I’m the best to answer this question so let’s also wait for others to chime in, but based on the last description I would expect the conv layers to depend on the channel ordering. If you are not keeping it consistent I would claim the use case could be seen as shuffling the color channels of input images randomly and expect the model to “reshuffle” it somehow.