It would depend on the use case and what kind of information is stored in each channel.
You could certainly try to reduce the channels to three, but would need to check, if you are losing valid information. Another approach would be to either replace the first layer with a new conv accepting more input channels, manipulating the kernels in the first layer and add new randomly initialized channels, or to add a complete new “0th” layer to transform the images to 3 channels (let’s call this a “trainable reduction”) using e.g. a 1x1 kernel.