You could treat the use case in the same way variable input shapes are treated in vision models, i.e. by adding an adaptive pooling layer before passing the activation to the linear layer.
This will make sure that the expected number of features are constant and you will be able to pass different sequence length.
Also, flatten the activation via x = x.view(x.size(0), -1).
You could try torch.nn.LazyConv2d to avoid the need to define the number of input channels.