Solved by ptrblck in post #2 
              
              
                I think it would be faster to reshape your input, so that your channels are stacked in the batch dimension. 
[batch_size, channels, h, w] would become [batch_size * channels, 1, h, w]. 
Then you could use a conv layer with in_channels=1 and out_channels=1 and reshape the output again. 
batch_size = …