Solved by ptrblck in post #9 
              
              
                Permute the input, so that the channel dimension is in dim1 and set the in_channels to the number of input channels, not the height.