Note that each kernel has the same number of channels as your input activation, which will be part of the summation, so you would need a depthwise convolution, where each filter acts on a single input channel.