How would the input channels be treated?
Would the linear layer perform an operation such as:
# input patch
patch = torch.randn(in_channels, height, width).view(-1)
lin = nn.Linear(in_channels * height * width, 1)
out = lin(patch)

If so, then this would be equal to a conv layer without overlapping windows, no?