Hi,
I am very new to transformers and I am working with FNet by Rishikksh20.
I was looking at some videos about transformers and I see that the input after embeddings is usually a matrix. However, in this implementation, there’s a Linear layer instead of 2d Convolution. So is the input usually a vector or is there a way I can input a matrix? I am doing this:
data_model = PositionalEncoding(data_model)
data_model = data_model.flatten()

Let me know if I’m doing it correctly.