Hey, guys. I am doing a sequence classification task using nn.TransformerEncoder(). Whose pipeline is similar to nn.LSTM().
I have tried several temporal features fusion methods:


Selecting the final outputs as the representation of the whole sequence.


Using an affine transformation to fuse these features.


Classifying the sequence frame by frame, and then select the max values to be the category of the whole sequence.


But, all these 3 methods got a terrible accuracy, only 25% for 4 categories classification. While using nn.LSTM with the last hidden state, I can achieve 83% accuracy easily. I tried plenty of hyperparameters of nn.TransformerEncoder(), but without any improvement for the accuracy.
Could you guys give me some practical advice? Thanks