Solved by Usama_Hasan in post #2 
              
              
                Hy Tom, 
Yes you can use rectangular convolution filter. 
# consider input (3,128,128)
torch.nn.Conv2d(3,64,(1,3),stride=2,groups=1)
torch.nn.Conv2d(64,128,(3,1),stride=2,groups=1)
# Then output after these convolutions will be (128,31,32)