You could use a grouped convolution, which would use a single filter for a group of input channel, and in the case of a depthwise convolution an own filter for each input channel.
Have a look at the docs 101 for more information.