Any number of dimensions are supported and the linear layer would use them as if you would loop through these dimensions as seen here:


lin = nn.Linear(10, 10)
x = torch.randn(2, 10, 10)
out = lin(x)
out2 = torch.stack([lin(x_.squeeze(1)) for x_ in x.split(1, dim=1)], dim=1)
print((out - out2).abs().max())
# > tensor(0., grad_fn=<MaxBackward1>)



and 3. yes, that’s correct.


nn.BatchNorm1d accepts inputs in [N, C, L], where C is optional. C is the channel dimension and num_features in the batchnorm layers will use either C or L depending on the input shape as described in the docs:





num_features – C from an expected input of size (N,C,L) or L from input of size (N,L)


The part in the Shape section is inconsistent, as it mentions the inputs as (N, C) or (N, C, L). A PR is welcome in case you want to fix it.


C denotes the channel dimension and thus the features in the batchnorm layer.




 denexo:

so C is both number of channels and number of features which is weird.


Why is this weird? This dimensions is used as the channel dimension in the input and named features in the normalization layer.



 denexo:

So perhaps num_features should be renamed to num_channels.


This might be a possibility for batchnorm, but wouldn’t match other norm layers, which do not necessarily use the channel dimension only.