Solved by smth in post #2 
              
              
                the right thing to do here is to have a nn.Conv2d(1, 1, (1, W)) and then fold the channels dim into batch: 
input = Variable(torch.randn(Batch, C_1, H, W))
m = nn.Conv2d(1, 1, (1, W))

output = m(input.view(Batch * C_1, 1, H, W)).view(Batch, C_1, H, 1)