I think you want to transpose the dimensions of your input tensor before and after (Linear — PyTorch 1.9.0 documentation 50 say it expect a Nx*xC_in tensor and you give him a 32x…x1 tensor)
Something like linear(x.transpose(1,3)).transpose(1,3) ?