You could try to move the batch dimension to the channel dimension and use a grouped convolution, where e.g. each kernel is only applied on a single input channel.
After the convolution is done you could reshape the output to the original shape.