Solved by ptrblck in post #2 
              
              
                If you don’t want to reduce the depth of your input volume, you could use e.g. a kernel size of 1 for this dimension (or another kernel size with the appropriate padding): 
encoder = nn.Sequential(
    nn.Conv3d(in_channels=3, out_channels=16, kernel_size=(1, 3, 3), stride=(1,2,2)), nn.ReLU(), nn.Ba…