Hi @lewtun @talolard ,
Thank you both for your thorough replies, it is greatly appreciated.
We will check out Longformer! Hopefully, BigBird 8 will be available through Huggingface in the near future as well!
I understand TF-IDF and WordPiece is not the same thing, just “hoped” there was some ranking of tokens involved there. I see now that ranking tokens from a score in WordPiece really doesn’t make that much sense.
We are considering not aggregating the tweets on user-level, our only problem is that our dataset is annotated on user-level (not collected by us), and classification is previously done on users in related research. If we convert it to tweet level, it would require a lot of hours to label these tweets manually or semi-automatic (as there are 10 million tweets).