I think it would be faster to reshape your input, so that your channels are stacked in the batch dimension.
[batch_size, channels, h, w] would become [batch_size * channels, 1, h, w].
Then you could use a conv layer with in_channels=1 and out_channels=1 and reshape the output again.
batch_size = 10
channels = 3
h, w = 24, 24
x = torch.randn(batch_size, channels, h, w)

conv = nn.Conv2d(1, 1, 4, 2, 1)
output = conv(x.view(-1, 1, h, w)).view(batch_size, channels, h//2, w//2)
print(output.shape)