It looks like you are working with some volumetric data based on the shape. You could apply some pooling, or another conv layer over the last two dimensions. I don’t think there is a “best” way and it will most likely depend on your model and use case.


Usually you would flatten the tensors to have the shape [batch_size, in_features]. However, nn.Linear layers also take arbitrary dimensions for inputs such as [batch_size, *, in_features] and apply the linear transformation on each sample in the asterisk dimensions.
Would you like to apply the linear layers separately on each of the inputs or would you like to concatenate the inputs in the feature dimension?